Skip to content
SHC Docs

Enable backups

This guide is for an estate where backup.repo and backup.password are already set but shc backup ls shows nothing worth restoring. It gets the first real archive into object storage, and it names the two traps that make this operation riskier than it looks.

Read section 4 before you run anything. On a daemon older than v0.0.18 the first capture of a database stack could leave that database read-only if it failed: an outage caused by taking a backup rather than by not having one. v0.0.18 closes that (ce 8111eb3), but the check is thirty seconds and the remedy is one command, so section 4 keeps both.

v0.0.18 also changes what a capture costs. shc backup create used to read the volumes with the stack running; ce 729a649 gives it a quiesce bracket, so every capture now pauses the stack for the duration of the read. Section 3 and the Expected outage table in section 5 are written for that behaviour.

Source of truth: modules/backup/encryption.go (which engine covers which archive), modules/backup/service.go (CreateBackup, the encrypting path), modules/backup/phases.go (the orchestrated path), subsystems/impl/backup_scheduler.go (what a schedule actually fires).


Measured read-only on the leaders (promanager 204.168.129.34, bitspur 37.27.215.210). Both estates are configured identically.

★ THE DEPLOYMENT COUNTS BELOW ARE STALE. Re-derive before you use them. This section originally recorded an identical 12 deployments on both estates. They are not identical, and neither number should be trusted now: the two estates have diverged since 2026-08-04, and bitspur in particular carries substantially more than 12. Run

Terminal window
shc ls --all-tenants

on each leader and work from what it prints. Everything downstream that is sized by deployment count (the “12 captures” in section 5.4, the “twelve snapshots” check in section 6, and any estimate of how long the sequence takes) is only as good as that number.

Section 5’s per-stack sequence enumerates the promanager estate only. It is a worked example, not a fleet inventory: run it against the output of shc ls --all-tenants on the estate in front of you, and expect bitspur to need stacks that are not named there.

Factpromanagerbitspur
backup.repos3:https://sos-de-fra-1.exo.io/shc-backup-promanager-hel1…/shc-backup-bitspur-hel1
backup.passwordref+vault://shc/system/backup/passwordsame
backup.aws_access_key_id / …secret_access_keyvault refs, both setsame
shc backup ls5 rows, every one pre-uninstall, every one ENCRYPTION none, newest 2026-07-30no backups
shc backup schedule lsno backup schedulesno backup schedules
shc doctor run -c backupinfo: key resolvesinfo: key resolves
Bucket reachability from the leaderHTTP 403 AccessDenied, TLS in 0.11sHTTP 403 AccessDenied
Live deployments (STALE: re-derive)12 as of 2026-08-04 (8 in corp/prod, 4 in master/default)NOT 12: bitspur has grown well past promanager; re-derive with shc ls --all-tenants
Bytes under mounts/~5.7 GB~6.1 GB
Free space on /var/lib/shc556 GB of 601 GB556 GB of 601 GB

Every one of promanager’s five archives names a stack that no longer exists (vaultprobe, s3front, s3proof-db, s3proof-nc). Neither estate held a single archive for any of its live deployments.

shc schedule ls is a different subsystem: per-stack app job crons. The backup schedule table is shc backup schedule ls. Both are empty; check the second one when you are asking about backups.


Four things could have caused this. Three were ruled out by measurement.

Configuration missing: no. All four backup.* keys are set at global scope on both estates.

Vault reference unresolvable, or vault locked: no. shc doctor run -c backup returns the branch that names the repository, and DiagnoseBackupEncryption only reaches that branch after the config loader has returned both a repository and a password; a loader error or a locked vault takes an error: branch instead. For contrast, a daemon with nothing configured reports [fail] no backup key configured, shc backup create will refuse. Both estates report the configured branch.

Object storage unreachable, or the bucket absent: no. An unauthenticated GET of the bucket from each leader returns 403 AccessDenied over TLS in about 0.1 seconds. The same request against a deliberately nonexistent bucket name returns 404 NoSuchBucket, so the 403 is positive evidence that the bucket exists and that egress and TLS work.

Nothing refuses. No schedule was ever registered, and shc backup create was never run. That is the whole cause.

The five promanager archives are not backups anyone took. They are the pre-uninstall safety archives the orchestrated engine writes automatically when a stack is removed, which is why each names a deleted stack, and why bitspur, which has never uninstalled anything, has none.

One vestige worth knowing about: /var/lib/shc/backups/ on both estates contains an empty restic repository (config, data/, index/, keys/, snapshots/, created 2026-07-27) with zero snapshots. It is not the configured repository and nothing writes to it. Leave it alone.


Since 0.1.28 SHC has one backup engine (FZ-7061). shc backup create, each member of --all-connected / --with-dependants, the backup schedule cron, the pre-uninstall safety backup, the pre-restore and pre-rollback snapshots, clone and move all run the same capture: freeze, copy, .bak, then FINALIZE seals that .bak into backup.repo, reads it back from the repository at its byte size, and deletes the local copy. The row reads encryption=restic and location=restic:<repo>:<snapshot>.

every backup
EncryptedYes, by sealing. A scope with no backup.repo or no backup.password is refused before anything is captured (X400486BKPNRP; a scheduled fire is skipped with X400079BKPSKD)
Destinationbackup.repo (your bucket), redirectable at every config scope
A seal that failsfails the run (X500480BKPNSL; scheduled X500474BKPSRQ) and discards the plaintext with the run’s staging. Nothing rests on local disk
Stack outagepause, escalating to a full stop past timeouts.quiesce_hold (300s); the retake is a delta over the copy already made. --fail aborts at the hold instead, with the stack running

shc uninstall on a scope with no repository therefore stops at its safety backup: configure one, or pass --no-backup to tear down without an archive. A restore that takes a rollback point stops the same way (--no-rollback). Archives shc backup create wrote up to 0.1.27 are restic tree snapshots of the deployment directory; they stay restorable, exportable and verifiable.

What the key does not cover. Three artifacts stay plaintext on purpose, and each is named on every branch of the backup.encryption doctor probe:

ArtifactWritten byWhy it is not encrypted
{state_home}/exports/<id>.bkgbackup create --all-connected / --with-dependants, backup groupIt bundles members by DECRYPTING them back out, because it exists to be read by shc recover --file on a cluster with a different key. Owner-only (0600 in a 0700 dir) and never pruned by retention; delete it yourself. For clusters that share a backup.password, shc recover --consistency-id <id> reads the same group straight out of the repository and produces no plaintext at all.
pre-move archiveclone/moveA kept pre-move archive rests into the stack’s backups dir as an ordinary backup, labelled as pre-move, and from then on the normal retention applies to it. The capture stages under <staging_dir>/clone/<operation id> and that directory is removed when the operation ends.
backup create --file / backup export outputyouYou asked for a portable archive at a path you named. A --file create writes only that file: it records no backup and takes no repository snapshot, with a repository or without one. The archive is plaintext on disk from the start of the capture: it grows at <staging_dir>/export/<id>.bak (0600 in a 0700 dir) while the capture runs, then waits there on the node that staged it until the CLI takes it, and the take unlinks it. One the CLI never takes (an interrupted or timed-out create) is removed an hour after it was staged by backup_recovery_sweep, the node-local sweep that runs every minute on every node. One that never finishes staging is removed once it has been staging longer than timeouts.backup_stack (6h unless you set it), the budget every capture runs under. The first sweep after a daemon restart removes whatever the previous daemon left there. Keep SHC_STAGING_DIR on each node’s own disk (one staging dir per node).

Every new backup is encrypted, or it does not exist. The seal is not conditional any more: no repository, no key, an unreachable repository, or a seal whose in-repo listing cannot be verified all fail the run with the reason named, and the seal waits up to five minutes for a lock another restic operation holds on the repository, such as a prune’s, before it gives up. A schedule created while backup.repo is set and backup.password is not is refused outright (X400077BKPSCD).

Plaintext archives from before 0.1.28 may still stand in {state_home}/backups. An unsealed orchestrated .bak is a plain !<arch> container holding manifest.json plus an uncompressed tar, with no key needed to read either. On a real app it carries app/vars.yaml, the deployment’s install-time --var values, in the clear, and the app’s own credential files verbatim: a Nextcloud config.php dbpassword, a frappe site_config.json encryption_key, a postgres converter’s pg_dumpall role lines. The one sealed member is secrets.vlt. Treat {state_home}/backups as secret material, and prune what is already off-site.

ce modules/backup/artifact_bytes_test.go plants a credential-shaped canary in the payload, runs each orchestrated caller, and reads every byte of every artifact root looking for it, with a paired positive control that requires the canary in a caller-named --file archive, so the absence assertions cannot pass over a capture that captured nothing. A second tier drives a real restic binary against a real repository and searches the repository’s own bytes.

The consistency trade-off, narrowed at v0.0.18

Section titled “The consistency trade-off, narrowed at v0.0.18”

This records how the restic-direct engine, removed in 0.1.28, came to freeze.

Both engines now freeze. Until v0.0.18 the encrypting engine took no pause and no stop: CreateBackup held no quiesce bracket at all, only app-level pre_backup hooks, confirmed by polling docker inspect 300 times across a capture with zero paused samples. The engine that protected the data did not encrypt it and the engine that encrypted it did not protect it, and shc backup ls rendered both identically.

ce 729a649 closes that half. CreateBackup now holds quiesce.go bracketPause (the same primitive the orchestrated flow and the routed single-mount RPC already shared, not a second copy), in the order

pre_backup (stack live) → pre_pause → PAUSE → stamp + stage + restic
→ UNPAUSE → post_pause → post_backup

Three properties are worth knowing before you run it:

  • The release is armed by defer in the same statement group that acquires the freeze, so a staging error, a restic failure, a cancelled request or a panic all still unpause the stack.
  • Only the read is inside the window. The whole manifest is built with the stack live; under the freeze SHC stamps the consistency claim, writes the few-KB manifest and runs restic.
  • A failed freeze does not abort the capture. It stamps the archive live + not_quiesced and proceeds, so a stack SHC cannot pause still gets a backup and the row says which kind you got. Whether the restic path should instead escalate to a full stop the way pause_hold.go does is an open owner call.

So a per-stack shc backup create is now encrypted and a real point in time, at the cost of a pause it did not used to take. Each stack is still its own point in time. --all-connected gives a true consistency group and is the only whole-product restore for affine, huly and keygen (whose own backup.yaml files say so), but it is plaintext and on-box. Section 5 takes the encrypted per-stack route and section 5.4 explains how to order it so that skew degrades gracefully.

(--all-connected needs a non-empty integration graph. On a stack with no connections it refuses with X400403BKPFAL … at least one path is required, which is expected rather than a fault.)

An app bound to a shared postgres, mysql or mariadb engine keeps its rows on that engine, not in its own volumes. Its per-stack backup now carries them anyway: the backup dumps the one database the app’s plug names through the engine’s exporter, and no other database of that engine. shc backup ls shows each dump and its size under database dumps. A lone restore imports the dump onto the engine the app is bound to, creating the database if the engine lacks it, and leaves every other database alone. A failed dump fails the backup with X500848BKPDBD. The engine’s own backup is unchanged, and an --all-connected run that captures the engine does not dump the app’s database a second time. shc backup restore --paths database/<plug> restores the database alone.


4. The trap: a failed capture could leave a database read-only

Section titled “4. The trap: a failed capture could leave a database read-only”

FIXED IN v0.0.18 (ce 8111eb3). If your daemon is older, this is live.

The four postgres stacks per estate declare a hook pair in app.yaml:

pre_backup: [quiesce] # ALTER SYSTEM SET default_transaction_read_only = on; CHECKPOINT
post_backup: [resume] # ALTER SYSTEM SET default_transaction_read_only = off

That is how a capture stays safe for a database: postgres goes read-only and checkpoints, so the filesystem read is consistent. The quiesce is persistent: postgres writes the switch into postgresql.auto.conf, so it outlives a container restart and a reboot. The only thing that takes it back off is the post_backup resume hook. Every database SHC packages does something in this family: supabase the same ALTER SYSTEM, mysql SET GLOBAL super_read_only = ON, mariadb SET GLOBAL read_only = ON, nextcloud maintenance mode.

Note the two mysql-family spellings are not interchangeable, and this page said they were until 2026-08-08. super_read_only is a MySQL 5.7+ / Percona variable; MariaDB has never implemented it (MDEV-9458 is closed WON’T FIX), and a mariadb server answers ERROR 1193 (HY000): Unknown system variable. Their persistence also differs from postgres’s: SET GLOBAL is not written to disk, so a mysql-family gate that the resume hook fails to clear survives until the container next restarts: long enough to matter, not forever.

What was wrong before v0.0.18: the undo was an ordinary statement on the happy path. On the restic path runBackupPostJobs was called on the success path only, with eight early returns between it and the pre_backup call; on the orchestrated path it was the last statement of FINALIZE’s success branch, so every failure return and the whole abort() path skipped it too. One of the eight is EnsureInitialized, the first-ever restic init against the bucket, the single most likely thing to fail on a first run, and it sat inside the read-only window. So an operator’s first backup attempt, failing on a typo’d S3 key, a wrong region or blocked egress, armed the read-only switch and then left. Measured on a disposable daemon against the pre-fix tree: INSERT failed with cannot execute INSERT in a read-only transaction, and a container restart did not clear it.

How v0.0.18 closes it: the undo is bound to the thing that armed it rather than added at another call site. openBackupBracket fires pre_backup and returns a closer the caller defers at the arming site, so every exit path (early return, error, cancellation, panic) pays it. It is once-only (the happy-path call keeps its original ordering and the hook never double-fires), a no-op when nothing was armed, and its context deliberately drops cancellation: half the exits it backstops are a cancelled or deadline-exceeded request, and an undo bound to the caller’s context would decline to run in exactly the cases that need it most.

The undo’s own failure is now reportable, which it was not: the seam was void, so the backup module had no way to say a resume hook had failed. It surfaces as X500459BKPUNQ (500, “backup of <stack> could not un-quiesce the stack, it may still be read-only”). It never replaces the capture error that got you there. If you see that code, run the check below: it is the one remaining path to a stuck read-only database, and it is now loud instead of silent.

4a. A quiesce is a HARD ERROR to your applications, not latency

Section titled “4a. A quiesce is a HARD ERROR to your applications, not latency”

Whatever the daemon does with the outage (a pause, or the full stop it escalates to past timeouts.quiesce_hold), the app-level pre_backup quiesce that runs before it does not make clients wait. It makes them fail.

ALTER SYSTEM SET default_transaction_read_only = on does not queue writes. A connection that is already open and takes a write lock during the window gets an error back:

ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction
ERROR: cannot execute INSERT in a read-only transaction

The read-only gates on mysql/mariadb and nextcloud’s maintenance mode behave the same way in their own vocabulary. So a scheduled backup of a database that other stacks talk to is, for its duration, an outage those stacks see as 5xx, not a slow patch, and an application whose retry policy assumes transient latency will surface it to users rather than ride it out.

Two things follow for how you schedule and monitor:

  • Schedule database captures in a window you would accept a short failure in, and treat a health check over a dependent app the same way you would around a restart, not around a slow request.
  • If an app must ride a backup without erroring, it needs a retry that treats read-only transaction as retryable, or the stack needs to be captured through a path that does not arm the app-level quiesce at all (leaving a mount out of include in the stack’s backup.yaml takes it out of the capture; it does not remove the hook).

This is a property of the quiesce mechanism, not a defect in it: a read taken over a database that is still accepting writes is the untrusted archive the whole bracket exists to prevent. It is documented here because the failure is reported to the application, which means it lands in the app’s logs and not in shc’s, so an operator correlating a 5xx spike with a backup window has no other place to learn the connection.


Three consequences for the procedure. The first two are cheap insurance rather than strict necessity at v0.0.18, and remain mandatory on an older daemon:

  1. Prove the object-storage path on a stack with no pre_backup hook first. Only openobserve and the four postgres stacks have hooks, and only the postgres pair leaves sticky state.
  2. Capture the databases one at a time, checking each afterwards.
  3. Know the remedy before you need it.
Terminal window
# check (safe to run any time)
sudo docker exec master_default_postgres-postgres-1 \
psql -U postgres -tAc "SHOW default_transaction_read_only" # want: off
# remedy if it says "on"
shc job run resume -t master -s postgres # or:
shc job run clear-read-only -t master -s postgres

Baseline on 2026-08-04: all eight postgres instances across both estates reported off. Nothing is currently stuck.


Run these on the leader. Substitute the tenant and environment shown: the corp deployments live in environment prod, the master ones in default.

--fail tells a capture to abort rather than escalate to stopping a stack. As of v0.0.18 the encrypting path holds a pause of its own, so this is no longer merely insurance against landing on the orchestrated path by accident; keep it on. Note the asymmetry: the restic path does not escalate a failed freeze to a stop at all (it stamps not_quiesced and proceeds), so --fail bites on the orchestrated path and on the hold deadline, not on a pause SHC simply could not take.

5.1 Preflight: read-only, about 30 seconds

Section titled “5.1 Preflight: read-only, about 30 seconds”
Terminal window
shc doctor run -c backup # expect: info, naming your bucket
shc backup schedule ls # expect: no backup schedules
shc vault status # must be unlocked
shc ls --all-tenants # ENUMERATE the deployments: this is the count everything below is sized by
df -h /var/lib/shc # headroom for the staging copy

If doctor reports an error: branch, stop: the key does not resolve and every capture will refuse. Unlock the vault (shc vault -p …) and re-check.

5.2 Prove the bucket path on a hook-free stack, under a minute

Section titled “5.2 Prove the bucket path on a hook-free stack, under a minute”

keycloak has no mounts/ directory and no pre_backup hook, so this is the cheapest possible probe of credentials, restic init and the first snapshot. It produces a small metadata-only archive; that is expected and still proves the path. (Verified on a throwaway stack with no volumes: the row came back encryption=restic.)

Keycloak keeps no data of its own here: its realms live in the keycloak database inside the master/postgres stack (KC_DB_URL points at default.postgres.postgres:5432/keycloak), so section 5.4 is what actually protects your realms and clients.

Terminal window
shc backup create -t master -s keycloak --name enable-backups-probe --fail

Then confirm the row before going further:

Terminal window
shc backup ls -t master | head -20

encryption must read restic and location must start with restic:. If it does not, stop and fix that first; everything below depends on it.

Then push real bytes through, still hook-free:

Terminal window
shc backup create -t master -s traefik --name enable-backups --fail # ~270 MB

Neither belongs to an app-plus-database group, so order does not matter. openobserve flushes in pre_backup and leaves no sticky state.

Terminal window
shc backup create -t corp -e prod -s license-broker --name enable-backups --fail
shc backup create -t master -s openobserve --name enable-backups --fail # ~5 GB, the long one

5.4 The three groups: database first, then its app

Section titled “5.4 The three groups: database first, then its app”

Capture each database before the stacks that reference it. With independent snapshots, a database older than its blob store yields orphan blobs, which are harmless; a database newer than its blob store yields dangling references, which are broken attachments. Older-database is the direction that degrades gracefully.

Each database carries the read-only hook from section 4, so do these one at a time and check the flag cleared before moving on.

Each app’s own capture below also dumps its database from the engine (section 3), so an app restored alone gets its rows back. The engine captures stay the copy of the whole engine.

huly keeps its rows in huly-db and its uploaded files in the hosted provider’s bucket, reached through an external-s3 front. Its own volumes hold the derived fulltext index and queue offsets. Back up the hosted bucket separately after the database capture. See Object storage for apps for the connection setup. affine and keygen keep their blobs in their own volumes, so each is a plain pair.

Terminal window
# --- keycloak's database (keycloak itself was captured in 5.2) ---
shc backup create -t master -s postgres --name enable-backups --fail
sudo docker exec master_default_postgres-postgres-1 \
psql -U postgres -tAc "SHOW default_transaction_read_only" # want: off
# --- affine ---
shc backup create -t corp -e prod -s affine-db --name enable-backups --fail
sudo docker exec corp_prod_affine_db-postgres-1 \
psql -U postgres -tAc "SHOW default_transaction_read_only" # want: off
shc backup create -t corp -e prod -s affine --name enable-backups --fail
# --- keygen ---
shc backup create -t corp -e prod -s keygen-db --name enable-backups --fail
sudo docker exec corp_prod_keygen_db-postgres-1 \
psql -U postgres -tAc "SHOW default_transaction_read_only" # want: off
shc backup create -t corp -e prod -s keygen --name enable-backups --fail
# --- huly: database, then the app ---
shc backup create -t corp -e prod -s huly-db --name enable-backups --fail
sudo docker exec corp_prod_huly_db-postgres-1 \
psql -U postgres -tAc "SHOW default_transaction_read_only" # want: off
shc backup create -t corp -e prod -s huly --name enable-backups --fail # ~600 MB

This is not a fleet inventory. Reconcile this list against shc ls --all-tenants on the estate you are working on, and add a capture for every deployment it names that does not appear above. Bitspur in particular needs more than these.

StepOutage
Every shc backup create abovenone: no pause, no stop, no restart
A postgres stack, while its capture runswrites rejected (read-only), reads unaffected; seconds to minutes with the data sizes above
A postgres stack, if its capture failswrites stay rejected until you run shc job run resume -t master -s postgres; see section 4
The restore test in section 6the stack is stopped for the replay

Exit code 0 is not verification. This project has shipped a truncated dump that returned 0, and there are two specific ways a backup can look healthy here and not be.

shc backup ls showing a completed row is not proof the archive exists. Demonstrated: after deleting a snapshot from the repository behind the daemon’s back, shc backup ls still listed the row as completed with encryption=restic. Only SIZE BYTES dropped to 0. shc backup sync did not reap it either. Do not treat the list as evidence.

Use these instead, in order of strength.

Check the posture per row. encryption must be restic and location must start with restic:. A row reading none came from the plaintext engine and is sitting on the same disk as the data.

Terminal window
shc backup ls --all-tenants

Confirm the snapshot is present in the repository. This asks the repository, not the index, and it genuinely fails when the snapshot is gone, verified by deleting one and watching it return X400405BKPINV while an intact sibling still passed.

Terminal window
shc backup verify <backup-id>

Count the objects in the bucket. Ground truth, independent of SHC. Run on the leader; the credentials are the same backup.aws_* values the daemon uses.

Terminal window
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
export RESTIC_REPOSITORY='s3:https://sos-de-fra-1.exo.io/shc-backup-<estate>-hel1'
export RESTIC_PASSWORD=…
restic snapshots # expect one snapshot per stack you captured
restic stats latest # non-zero size
restic check # repository integrity; verify only probes membership

You should get one snapshot per stack you captured; compare against your own count from shc ls --all-tenants, not against this guide’s twelve. Fewer means a capture failed quietly.

Restore-test one stack. The only check that proves the bytes are usable. Pick something you can afford to stop, and note the stack is stopped for the replay. Measured on a small stack: 1.79 seconds stopped, plus 364 ms paused while the pre-restore rollback archive was taken. Real stacks scale with the data being replayed, not with the archive size.

Terminal window
shc backup restore <backup-id> -s <stack> --yes

Then compare a checksum inside the container against the value from before the restore. On the throwaway proof both a text sentinel and a 64 KiB binary blob came back with identical MD5 sums after being deliberately corrupted, so the replay is real rather than a no-op.

A restore takes its own pre-restore rollback archive first, and that one is written by the plaintext engine to local disk. Expect a new encryption=none row to appear; that is the mechanism, not a fault.

Bringing back a deployment you uninstalled

Section titled “Bringing back a deployment you uninstalled”

An uninstall takes a pre-uninstall safety backup, then drops the deployment’s config and vault secrets. A database or bucket it held on a provider stays there unless its plug’s cleanup was set, and --no-backup keeps its block volumes. That leftover data guards against an accidental loss. It is not where a later install resumes: an install under the same name is a new deployment, and it refuses while any of it is there, naming each leftover (X409440LEFTOV, X409440VOLKPL). shc install --from-backup is a new deployment as well and refuses the same way.

The way back is shc recover with that safety backup. It brings the deployment back over the leftovers, with its data and its vault secrets:

Terminal window
shc backup ls # the stack's newest pre-uninstall row; LOCATION is the archive
shc recover -b /var/lib/shc/backups/<tenant>/<stack>/<id>.bak

To start fresh instead, delete the leftovers the refusal names, then install. Set --var <plug>.cleanup=true, on the install or a later shc update, and the uninstall drops them itself.

Everything above is a one-off an operator has to remember to run. Ruling B3 (docs/decisions/2026-08-14-backup-rulings.md) makes the deep tier standing: the package ships /usr/lib/shc/scripts/backup-verify-sweep.sh plus a shc-backup-verify.timer/.service pair that runs a weekly, budget-capped shc backup verify --deep of the newest good archive per deployment: newest completed row whose location is a restic snapshot. Deployments are visited stalest-turn-first (then smallest-first): a turn is the last time the sweep started a deployment’s read, proven or cut short, kept in /var/lib/shc/backup-verify/last-attempted.tsv beside the proofs in last-proven.tsv. A budget that covers only part of the estate rotates coverage across weeks, and an archive the budget cuts off goes to the back instead of starving the tail.

The package arms the timer on every node, once. The sweep runs on the raft leader and a follower’s fire skips before it reads anything, so a leader move never leaves the estate unswept. To keep one install out, set SHC_BACKUP_VERIFY=off when the package is installed; to turn it off later, systemctl disable --now shc-backup-verify.timer, which an upgrade does not undo.

shc backup schedule ls on the leader ends with when the sweep last ran and what it found, and shc doctor run --category backup checks it as backup.verify_sweep. Each node keeps the report of the sweeps it ran, so the check asks every node for its report and reads the newest one, and every node gives the same verdict whichever one leads. It fails when that sweep disproved an archive, left one unproven for more than 15 days, or picked archives and proved none of them with no budget cut to blame, and warns when the estate has held sealed backups for more than eight days and no node holds a sweep or the newest one is older than that.

Terminal window
$ shc backup schedule ls
...
verify sweep: enabled on shc01, last ran 2026-10-10T12:14:03Z: 12 deployments: 11 proven, 0 disproven, ...

Each run prints a per-deployment verdict table (PROVEN / DISPROVEN / UNPROVEN when the budget ran out on it / UNVERIFIABLE for plaintext-only deployments and for a read the daemon could not finish), keeps the latest copy at /var/lib/shc/backup-verify/last-report.txt, and exits non-zero on any DISPROVEN row, which parks the unit in systemctl --failed until a later sweep passes. It also exits non-zero when it picked archives and proved none of them with no budget cut to blame (every pick ORPHANED or UNVERIFIABLE), and before it reads anything when a row of shc ls --all-tenants -o json has an empty tenant, stack or environment. Fires Saturday 12:00 UTC, the quiet half of the clock, hours from both estates’ 00:45–06:50 capture window.

The budget is the only bound. There is no per-archive size cap: every pick gets the deep read-back whatever it weighs, because a deep verify streams and costs the daemon O(1) memory in the archive. Measured against a real repository, deep reads of 8 GB, 4 GB and 2 GB archives held shcd at 43 MB peak RSS; the archive-scale memory is the restic child, which is flat in archive size and exits between verifies. What an archive costs is time and egress, which the wall-clock budget bounds (3600 s by default). A pick the budget runs out on is UNPROVEN with X409080VFYUNP, naming its size, the seconds its read ran and its last proof, and doctor fails once one stays unproven past 15 days. An estate with archives too big for the hour raises the budget, and the unit’s timeout with it:

Terminal window
$ systemctl edit shc-backup-verify.service
[Service]
Environment=SHC_BACKUP_VERIFY_BUDGET_SECONDS=14400
TimeoutStartSec=6h

On an estate with more than one location, each location label carries its own off-site repository, and the archives of a deployment land in the repo of the location its node is in:

Terminal window
shc label set location=hetzner/hel1 --backup-repo s3:https://fsn1.your-objectstorage.com/backups-hel1
shc label set location=hetzner/fsn1 --backup-repo s3:https://hel1.your-objectstorage.com/backups-fsn1

Put each location’s repo outside that location, so losing the location never loses its backups. The order that decides where an archive goes is: a backup.repo set at a tenant, environment or stack scope; then the repo of the deployment’s location label; then the global backup.repo. A location repo uses the same backup.password and the same backup.aws_* credentials as the global one, so an archive that was written before a move stays restorable from the repo its backup row names.

backup.repo may be a path, such as /var/backups/shc/repo. On a cluster of more than one node that is one repository per node, unless every node mounts the same storage there: a backup is on the disk of the node that took it and is lost with that node. shc config set backup.repo <path> says so when it is set, and shc backup ls then shows a NODE column naming the node that holds each archive (node and node_id in -o json).

The backup row is cluster-wide, so its id works from any node:

  • shc backup verify <id> typed on another node runs on the node that holds the archive.
  • shc backup export <id> fetches the archive from that node.
  • shc backup restore --id <id> runs on the node that renders the deployment. When that is not the node that holds the archive, it fetches the archive from the holder first, before anything stops, and replays the copy. The copy goes to a spool file on the restoring node and is removed when the restore is over, so that node needs free disk for the archive.

A holder that does not answer is the one refusal: X409732BKPELS, before anything stops, naming the node and the path. Before this, another node opened its own repository at the path and answered restic’s wrong password or no key found (SHC-1732).

backup.repo may name an s3 connection and a bucket in it, in place of a URL and a key in backup.aws_*:

Terminal window
shc connection connect backblaze --type s3 --scope system -t master \
--cred s3_endpoint=s3.us-west-004.backblazeb2.com --cred region=us-west-004 \
--cred access_key=<key id> --cred secret_key=- --value-file ./b2.key
shc connection grant backblaze -t master --capability s3 --grantee corp \
--boundary '{"buckets":["shc-backup-example"]}'
shc config set --scope global backup.repo @backblaze/shc-backup-example

The form is @<connection>/<bucket>[/<path>]. It works at every scope backup.repo is read at, and as a location label’s --backup-repo. The bucket must exist: nothing here creates one.

The key never sits in config. It comes through the s3 grant of the tenant whose deployment is backed up (the baseline tenant for a scope with none), and that grant’s --boundary must name the bucket. A tenant with no grant is refused X403910CONGRT. A grant that does not name the bucket, an unscoped one included, is refused X403925CONBKT. The connection must be of type s3.

Which key restic is handed depends on the connection’s own key:

  • Where the endpoint can mint a key for one bucket and the connection’s key is allowed to (Backblaze B2 with writeKeys, listKeys and deleteKeys, or Exoscale), SHC mints one key per tenant, restricted to that bucket, and the connection’s key is not used for the backup. shc connection revoke-grant deletes the minted key.
  • Where it cannot (a B2 key restricted to one bucket, or a store with no per-bucket keys such as Hetzner Object Storage), restic is handed the connection’s own key. It stays in the daemon, and only a bucket the grant names is opened with it. Prefer a key that is itself restricted to the bucket.

Archive rows, shc backup ls, shc backup relocate and the doctor all read the repository as its URL, s3:https://<endpoint>/<bucket>[/<path>]. shc connection rm is refused while a backup.repo names the connection.

The off-provider mirror in 6b reads this form too: the daemon resolves the bucket URL and the key the grant yields, and hands them to rclone itself.

6b. The second copy: the off-provider mirror

Section titled “6b. The second copy: the off-provider mirror”

A verified archive in one bucket is still one bucket. backup.repo is a single global value: one provider, one credential, holding every archive the estate has. The exposure that matters is “one repo, no second copy”, and the package closes it with shc backup mirror and a shc-backup-repo-mirror.timer/.service pair that runs it: a nightly, append-only, file-level rclone copy of the whole restic repository onto a SECOND provider under a SECOND credential. The daemon runs the copy. It reads each repository and its key the way restic is handed them, a connection-named backup.repo included, and gives them to rclone in its environment, so no key passes through the CLI or a shell.

Configure the destination once, in the vault. The daemon derives the S3 endpoint, region and rclone provider from each URL (a connection’s own region wins):

Terminal window
shc vault set backup/copy_repo s3:https://<host>/<bucket>
shc vault set backup/copy_aws_access_key_id <key>
shc vault set backup/copy_aws_secret_access_key <secret>

Each location label that names a --backup-repo gets its own copy on the same second provider, at <backup/copy_repo>/locations/<provider>/<location> (_ stands in for a location with no provider), with the same order and the same floor as the estate repo. One location that fails its copy fails the run, and the others are still copied. bitspur’s second provider is Backblaze B2 (SHC-1069).

That declaration is the whole arming. The package enables shc-backup-repo-mirror.timer on every node when it installs, and each fire decides for itself: a node that is not the raft leader says it skipped and exits 0, and so does a leader whose vault holds no backup/copy_repo. A leader move therefore needs nothing re-armed, and an estate that never declares a second provider never writes to one. A locked vault, or a daemon that cannot say who the leader is, fails the run instead of skipping it.

The package arms the timer once per node and records that in /var/lib/shc/.backup-repo-mirror-armed, so systemctl disable --now shc-backup-repo-mirror.timer is the lasting opt-out: it sticks across upgrades and across a remove and reinstall. SHC_BACKUP_REPO_MIRROR=off in the environment of one install keeps that install from arming it.

It copies; it never syncs. rclone copy has no delete side, and neither the mirror nor its tests permit one (mirror_2977_test.go fails on any delete-capable verb or --delete* flag reaching the code). That is the single most valuable property here: an accidental restic forget --prune against the primary cannot propagate into the copy. The destination therefore grows monotonically; put a noncurrent-version lifecycle on the bucket to bound the versioned half, and treat reclaiming pruned packs as a deliberate operator act rather than a schedule.

The copy order is load-bearing: config, keys, data/, index/, snapshots/. A repository read in the middle of a write must never see an index referencing a pack it does not have, so packs land before the index that names them and the index lands before the snapshots that name it: every intermediate state of the destination is a valid, merely older repository rather than a torn one. A run that fails part-way exits non-zero (parking the unit in systemctl --failed) and says so, and the destination is still openable: a file-level copy of a restic repo is a restic repo, so restic -r <copy> snapshots reads it with the same password.

shc backup mirror runs one mirror by hand and lists each repository with its status and the two snapshot counts its floor compared.

Fires 09:00 UTC, after the 00:45–06:50 capture window and far from the 02:00 retention prune, which takes an exclusive repository lock.

Why rclone and not restic copy. restic takes one set of S3 credentials per process (RESTIC_FROM_PASSWORD exists, RESTIC_FROM_AWS_ACCESS_KEY_ID does not), and the entire point of the destination is that it is a different provider under a different credential. rclone holds per-remote credentials, so it is the one tool in the payload that can span the two. It is a hard package dependency for the same reason jq is: the estates install with APT::Install-Recommends "false", so a recommend evaporates on exactly the posture SHC ships to.

This was a hand-dropped script, and that is the defect the packaging closes. Until 2026-08-20 the mirror existed only as /usr/local/sbin/shc-backup-repo-mirror.sh on one node of one estate. The estate rebuilt beside it carried the credentials and none of the machinery (no script, no unit, no rclone), so it ran on a single bucket, and nothing reported it: the timer that would have failed was the missing payload. A node that is rebuilt now gets the mirror from the package.


There IS an in-product way to schedule an encrypted, off-box backup: shc backup schedule set. Use it. You do not need a host cron.

CORRECTION. Until 2026-08-07 this section read “There is no in-product way to schedule an encrypted, off-box backup today” and described the schedule’s fires as “plaintext, on local disk, backup.repo ignored”. That was false and it contradicted section 3 of this same guide. ce de396a7 gave the orchestrated path a seal, and shc 27aff725e corrected section 3 to match, but missed this section, so the guide argued with itself for three days. The stale text sent operators to build a host systemd timer they do not need, around a mechanism that already works.

  • shc backup schedule set registers a schedule whose fires take the orchestrated path, which seals into backup.repo, encrypted, whenever a repository and password are configured. It writes a plaintext .bak and then FINALIZE seals it into the repository, verifies it readable there, and deletes the local copy. Section 3 has the mechanism; a fire that cannot seal is skipped or failed, never kept as local plaintext.
  • shc schedule add schedules an app-declared job from app.yaml, not an arbitrary command, so it cannot invoke shc backup create. That is unchanged, and it no longer matters: the backup schedule is the right surface, not a workaround for it.

What a schedule gets you that section 5’s manual sequence does not: it runs on its own. Since 0.1.28 a manual create and a fire run the same capture, with the same quiesce bracket that escalates to a stop past timeouts.quiesce_hold.

The two things to check before you rely on it, both of which section 3 establishes and neither of which a cron would fix:

  1. backup.repo and backup.password must both be set, or every fire is skipped with X400079BKPSKD naming what is missing. A schedule created with a repo but no password is refused outright (X400077BKPSCD).
  2. Confirm the first fire sealed: shc backup ls --all-tenants and check encryption=restic / location=restic:… on the new row, exactly as in section 6. A seal that failed reads failed with X500474BKPSRQ.

The schedule grammar:

Terminal window
shc backup schedule set "30 3 * * *" --scope stack -s <stack>
shc backup schedule set "@daily" --scope tenant
shc backup schedule ls
shc backup schedule unset --scope stack -s <stack>

Scopes are global, tenant, environment, stack, deployment. A schedule fires on the cluster leader.

★ DO NOT SCHEDULE AT 02:00, AND THIS IS NOT A STYLE PREFERENCE. retention_cleanup_cron is hard-coded to 0 2 * * * (ce modules/retention/crons.go, SpecRetentionCleanup) and is leader-only which is the same node your schedule fires on. Its sweep issues restic forget per snapshot and then one restic prune per repository, and prune repacks the repository under a repo-wide EXCLUSIVE lock (ce modules/backup/archive_delete.go). A capture that fires into that lock is in direct contention with the pruner for the whole repack.

The examples above use 30 3 for that reason. Keep the 02:00–03:00 hour clear of every schedule you register, and if you later see a bare 0 2 * * * in a diff, this paragraph is why it should not go back.

The expression grammar, exactly. The parser is cron.NewParser(Minute|Hour|Dom|Month|Dow|Descriptor) (ce modules/backup/scheduler.go, cronParser), which is the same parser the write path’s pre-flight and the live registration both use:

  • 5 fields, minute-first. There is no seconds field: a 6-field expression is rejected, not silently reinterpreted.
  • @ descriptors, the whole set: @yearly / @annually, @monthly, @weekly, @daily / @midnight, @hourly, and @every <duration> (a Go duration, e.g. @every 6h). Releases up to and including v0.0.27 named only @hourly / @daily / @weekly in --help; that was an incomplete summary, never a restriction the parser enforced.
  • An optional leading TZ=<zone> / CRON_TZ=<zone> prefix; without one the expression is interpreted in UTC via CE’s core/registry.ScheduleLocation().
  • No prose. "every 5 minutes" is not a schedule; it is a typo that gets refused.

Anything the parser rejects comes back as X400972INVCRN at schedule set: the schedule is not written, so a bad expression cannot become a dead row that never fires and never says so.

Three verb facts that are easy to get wrong:

  • There is no shc backup schedule get. The verbs are set, unset and ls (list is a cobra alias for ls). A shc.backup.schedule.get route does exist on the daemon, but it is an API surface (pkg/apiclient’s BackupScheduleGet, for the terraform provider) with no CLI spelling.
  • shc backup ls refuses positional arguments (cobra.NoArgs). Filter with --stack / --status / --all-tenants; shc backup ls <stack> is an error, not a filter.
  • unset converges, it does not stop the schedule immediately. The handler unregisters the live cron on the node that served the request and deletes the row. If a follower served it, the leader’s in-memory cron keeps firing until its own reload lands; the deleted row is what makes it converge on the next reload, leader transition, or restart. shc backup schedule ls reads the backup_schedule table, so it goes quiet immediately, before the leader does. If you need the fires to stop now, unset against the leader.

Two things to know before you pick the expression. A fire holding its freeze past timeouts.quiesce_hold (300 seconds) will stop the stack, retake the capture against the halted containers, and start it again: deliberate, because an archive nobody can trust is worse than a short outage; the stop is reported on the row and in StackStopSeconds. And section 4a applies in full: a scheduled capture of a database is a window in which dependent stacks see errors, not latency, so schedule it where you would accept that.

Until a fire is confirmed sealed, keep treating {state_home}/backups as secret material: an unsealed .bak is readable without a key (section 3).


Retention is ARMED, and it has a shipped default. Something prunes on its own: set the policy you actually want BEFORE your first capture, not after.

CORRECTION. This section previously said “nothing prunes on its own.” That was wrong. Verified in ce: modules/retention/crons.go registers retention_cleanup_cron on the spec 0 2 * * * as a leader-only cron, and the queue-side handler it enqueues is bound in daemon/services_seams_retention.go. It is wired end to end.

That 02:00 slot is spoken for, and it is why section 7 tells you to schedule elsewhere. The sweep’s restic prune takes a repo-wide exclusive lock, on the same node your backup schedule fires on. Leave the 02:00–03:00 hour to the pruner.

DefaultRetentionPolicy() (ce modules/backup/types.go, mirrored by retention.DefaultPolicy() in modules/retention/types.go) is what an unconfigured tenant gets:

keep
daily7
weekly4
monthly12
yearly5

So the cron begins acting at the first 02:00 after archives exist, against those numbers, whether or not you ever ran shc retention set. The bucket does not grow without bound, but neither does it keep what you assumed it would. Restic deduplicates, so a daily capture of the 5 GB openobserve stack costs far less than 5 GB per day; that is a reason to consider keeping more than the default, not a reason to leave it unset.

Set it explicitly before the first capture. An explicit policy is the difference between a retention window you chose and one you inherited, and the first 02:00 comes sooner than a review does.

Terminal window
shc retention ls # resolved daily/weekly/monthly/yearly counts
shc retention set --daily 7 --weekly 4 --monthly 12 --yearly 5 # the shipped default, stated
shc retention preview # what would be removed
shc retention apply # run it now rather than waiting for 02:00

shc retention ls shows the resolved counts, so run it once after setting to confirm the scope you set is the scope that resolves.

Two cautions carried over from how this repository handles keys and archives:

  • Rotating the restic password orphans every archive taken under the old one. There is no key-add verb. Decide the key once.
  • shc backup verify is a membership probe, not an integrity check. Restic’s own check is the whole-repository verification, and it takes an exclusive lock: run it outside a capture window.

Restic encrypts client-side under backup.password, and the two hel1 estates hold different passwords, so neither can read the other’s archives. That is not the same as isolation. Confidentiality is not availability: the S3 credential decides who can delete, and a delete needs no key at all.

Both hel1 estates were installed with the SAME Exoscale credential, and it was an Owner-role key, Exoscale’s unrestricted role, whose policy is literally {"default-service-strategy":"allow","services":{}}. Measured on 2026-08-04, that one key sitting in each estate’s vault could list, write and delete objects in the other estate’s backup bucket, and in bitspur-k8s-archive, a bucket that has nothing to do with either estate and holds a content-verified k8s dump set. Anything that could read either vault inherited all of it.

One S3 credential per estate, scoped to that estate’s own bucket. Not one per organisation, and never the org-Owner key. A backup credential’s whole job is s3:<endpoint>/<one-bucket>; anything wider is blast radius with no corresponding capability.

On Exoscale that is an IAM role whose policy denies by default and allows SOS only for one bucket, plus an API key bound to it. The expression variable is parameters.bucket; resources.sos.bucket and resources.bucket are both accepted by the API and then match nothing, which denies the credential its own bucket. Verify, do not assume:

Terminal window
cat > policy.json <<'JSON'
{
"default-service-strategy": "deny",
"services": {
"sos": {
"type": "rules",
"rules": [
{ "action": "allow", "expression": "parameters.bucket == 'shc-backup-<estate>-<region>'" }
]
}
}
}
JSON
exo iam role create shc-backup-<estate>-<region> --policy - < policy.json
exo iam api-key create shc-backup-<estate>-<region> shc-backup-<estate>-<region>

The secret is printed once, at creation.

There is no shc backup key set. shc backup key offers only ls / rm / rotate, all of which act on the restic repository password, not on the object-store credential, and rotate refuses when the current password is empty, so it is not an install path either. Do not reach for it: rotating the restic password orphans every archive taken under the old one and there is no restic key add verb (section 8).

The object-store credential has no special verb because it does not need one. It is an ordinary secret plus an ordinary config key:

Terminal window
# 1. the secret, into the vault at global scope (the vault must be unlocked)
shc vault set backup/aws_access_key_id '<EXO…>'
shc vault set backup/aws_secret_access_key '<secret>'
# 2. the config keys, as references, never as literals
shc config set backup.aws_access_key_id ref+vault://shc/system/backup/aws_access_key_id
shc config set backup.aws_secret_access_key ref+vault://shc/system/backup/aws_secret_access_key

Note the two spellings of the same thing: the vault key is backup/aws_access_key_id (what shc vault ls --scope global shows), and the config reference is ref+vault://shc/system/backup/aws_access_key_id (shc/system/ is the scope prefix in the ref grammar). shc vault get shc/system/backup/aws_access_key_id answers X404907SECNOF, which reads like a missing secret and is only a mis-spelled path.

Both keys resolve on demand through the secret dispatcher (LoadEnv in daemon/services_backup_config.go maps them onto AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for restic), so replacing a credential is a shc vault set and nothing else. No redeploy, no daemon restart. A locked vault makes every backup fail rather than fall back, which is intended, and means the credential swap wants the vault unlocked (shc vault -p).

Do not accept “the policy looks right”. Two probes settle it, and neither can destroy anything even if the policy is wrong:

  • DeleteObject on a key that does not exist in the other estate’s bucket. S3 DeleteObject is idempotent: an authorized delete of a missing key returns 204 and destroys nothing, an unauthorized one returns 403. Decisive, zero blast radius.
  • DeleteBucket without -r against a bucket that is not empty. An authorized call can only answer BucketNotEmpty; it can never delete.

Run every probe against a credential you know is unrestricted as well. A DENIED from a probe that never issued a request proves nothing: the first attempt at this measured exo storage delete without -f and recorded seven “allowed” verdicts that were actually confirmation prompts hitting EOF. Measured with the scoped keys in place:

operationown bucketother estate’s bucketbitspur-k8s-archive
ListObjectsallowed403403
PutObjectallowed403403
DeleteObjectallowed403403
DeleteBucketn/a403403

and the same matrix under the Owner key it replaces: allowed, allowed, allowed everywhere, with only BucketNotEmpty standing between it and a bucket delete.

A credential scoped to one bucket still has full delete rights inside that bucket. It has to, because restic forget/prune rewrite the repository. So per-estate scoping bounds the damage to one estate’s own archives; it does not make them immutable. If archives need to survive their own estate being compromised, that is object-lock / versioning on the bucket, which is a separate piece of work and is not wired.


10. Whose data is in this archive? The boundary audit

Section titled “10. Whose data is in this archive? The boundary audit”

The ruling. A stack’s backup captures the data that stack owns. It never follows an integration edge to capture data owned by the thing on the other side. Nextcloud must never back up its S3 bucket: that bucket is reached through an external-s3 edge and is not Nextcloud’s data. The provider owns its own durability story; crossing the edge duplicates another stack’s data under the wrong name and produces archives whose size is unrelated to the capturing stack’s real state.

What it cost when it was crossed. corp/hz-s3 is an external-s3 connector: services: {}, no volume, no database, no bucket of its own. With backup.buckets armed it pulled the Nextcloud bucket cloud-bitspur-com through a container into restic on 2026-08-13/14/15. Those captures blew the 30-minute converter budget, exceeded the ar(5) 10^10 size ceiling and were silently truncated to about 10%. An empty backup.buckets is the CORRECT state.

The report:

Terminal window
# the shipped catalogue
make backup/boundary-audit
# the catalogue AND a live estate, read-only
make backup/boundary-audit ESTATE=bitspur SUDO_CMD=sudo \
SSH_CMD='ssh -i ~/.ssh/estate_bitspur admin@<addr>'

It names each package OWNS-AND-CAPTURES / CROSSES-A-BOUNDARY / CAPTURES-NOTHING / AMBIGUOUS, and for a crossing prints the evidence, the variables that arm it and what removing it would cost.

Run the estate half, not just the catalogue. Whether a crossing is armed is decided by vars this deployment was installed or updated with, and the shipped package says nothing about it; the catalogue file was in its correct state throughout the hz-s3 incident. The estate half resolves each converter’s capture-set expression against the deployment’s own app/vars.yaml, and separately compares each stack’s archives against what that stack itself stores.

Two things it is not. It is not a CI gate and nothing fails because of it: it is something you run when you want to know. And it never disarms anything: every command it issues is a read, because turning off a running backup is a decision about what depends on it, not one a report gets to make.

backup.buckets being empty does not prove an external-s3 stack is disarmed. The converter falls back to the bucket serve pin, so --var bucket=<name>, set for access-control reasons, with the word backup nowhere in the command, arms a full-corpus capture by a second route. The report checks both.


11. What the estate says when none of this is armed

Section titled “11. What the estate says when none of this is armed”

Every section above describes something an operator has to do. Each of the five defects measured on 2026-08-20 was a section somebody had not done, and in every case the product held the information needed to notice and never looked. Four shc doctor lines close that, all under shc doctor run --category backup, and all of them are also emitted once a day by the backup_coverage_audit cron (08:00, leader-only) so they reach whatever notification rules the estate has rather than waiting to be pulled.

CheckAsksFires when
backup.encryptionis the archive encrypted with a key only you holdno key, half a key, or a key that will not resolve
backup.destinationcan a fire actually reach the repositorya cloud repo whose credentials are declared and do not resolve (error), or which needs credentials SHC has none configured for (warning)
backup.armedis what the estate declared actually armedbackup/copy_* in the vault with no mirror unit armed on the leader; schedules registered over an estate that has never captured anything
backup.coveragedoes each deployment have an archivenever protected, last attempt failed, stale, or object data in a bucket no capture reads
backup.orphansdo the ledger and the bytes still refer to each othera row that records no location; an archive file under the backups root belonging to no row

Two of these are worth reading together, because on 2026-08-20 they said opposite things about the same estate. backup.encryption reported healthy on promanager-new: backup.repo was an s3: URL and backup.password resolved out of the vault, which is exactly the question that check asks. The two backup.aws_* rows were ref+vault:// references to keys the vault had never held, so every scheduled fire was skipped before it opened a row: no archive, no failed row, no warning, five schedules, days of silence. backup.destination is the line that says so, and backup.encryption’s healthy line now points at it rather than reading as an all-clear.

backup.armed is the second-copy half of section 6b. An estate that has stored backup/copy_repo has declared it wants two providers; if nothing is armed to deliver that, the declaration reads as protection and is not. The check is silent on estates that never asked for a second copy, and says unknown rather than no when the vault is locked.

None of the four repairs anything. shc doctor run --fix will not arm a timer that starts paying a second object store, will not write a credential, and will not delete an archive; each hint names the capability and the operator performs it.


The replicas of a service bind the same mounts, so a backup of one has no copy to choose. backup.yaml takes no where: or prefer:: a file that sets either is refused (X400258BKPSEL) before anything is captured, and the refusal names the file and the key. Distinct copies of a service are a group of members.

A database that runs as a group of members keeps one dataset in every member, so the converter that captures it says from: <selector>, for example from: group=postgres, and each backup reads one member: the lowest-numbered member it matches that is ready and is not the writer, else the writer. The writer is the member the group-named alias with :1 picks, postgres’s role=primary, so from: group=postgres & role!=primary never reads the writer. A member whose pre_backup refuses, such as a standby that is behind, is passed over for the next one. The archive holds that one copy, never one per member or per location. With no member to read the backup is refused (X409895BKPGRP) and the refusal says why each member was passed over. An app’s from: uses only group, member, service and the labels the app writes itself, which hold on every estate (X400259BKPFRM otherwise). A restore writes the copy into the lowest-numbered member and empties the others, which clone from it. A group whose members hold distinct data names no from:, and every member is captured. See Database apps, HA by scale.

A replica keeps its mounts on the node that runs it. A backup captures them there, whichever node the backup itself runs on: the node a schedule fires on, the node a clone or move takes its backup on, or the node shc backup create reached. That node pauses the replica’s containers for its own capture and says whether the pause held, and the archive’s point_in_time and not_quiesced marker come from that answer. The node running the backup pauses nothing when no replica of the deployment runs on it.

A backup never completes without a replica’s data. It refuses, or ends failed, with X409899BKPRPL naming the replica, its node and why:

  • the replica’s node does not answer, or answers that it holds no such replica
  • the capture on that node fails
  • a service that is not stateful runs on several nodes and each holds its own copy of the same mount, and an archive carries one copy

A mount backup.yaml doesn’t list under include, whether it’s under keep or scratch, is not data the backup owes, so it never causes this refusal. shc backup create --file reads only the disk of the node it is written on and is refused when a replica’s data is on another.