Flow: backup and restore

Step-by-step trace of shc backup create and shc backup restore. Source of truth: modules/backup/service.go (CreateBackup — the restic pipeline), modules/backup/capture.go + modules/backup/orchestrator.go, and modules/backup/restore.go. Architecture overview: backup module.

Two backup destinations are supported, chosen by the caller:

• Restic repo — default. Content-addressed, deduplicated, encrypted. Data lives in the repo at config.backup.restic.repo_url and only a BackupCacheEntry is kept locally. Used for scheduled / retention- managed backups.
• Local .bak archive — explicit --output <path>. A single self-contained file on disk. Used for export / import style workflows, for clone/move, and for hand-off to an offline tenant.

Both paths share the same capture pipeline (steps 1-9 below) and diverge at step 10 on how the staging directory is finalized.

func (s *BackupService) Create(stack *DeploymentModel, request BackupCreate, output string) (*BackupModel, error)

• request.Environment — which environment of the stack.
• request.RunJobs — whether to run the pre_backup/post_backup backup config jobs (typically mysqldump, pg_dump, etc).
• request.CompressionLevel
• output — if set, use CreateLocalBackup instead of restic.

getRunningBackup(stack.Name, tenant) — if another backup is already in flight for this stack, return X400402BKPRUN. Prevents double-snapshots.

A backup_id is minted. The DB row carries status and started_at from the moment the backup starts — so the UI can show “backup in progress” even before data starts flowing.

createdSnapshotID and createdSnapshotRepo start as empty strings. They’re set immediately after the restic CreateSnapshot succeeds so the error handler (step 11) can forget an orphan snapshot if a later step fails.

restic.GetRepoPath(stack, env) resolves the per-stack repo path; EnsureInitialized(repo) is idempotent.

os.MkdirTemp(paths.StagingDir, ...). All captured data lands here first; restic snapshots this directory. Cleaned in the defer block regardless of success.

loadBackupConfig(stack) reads the stack’s backup.yaml (if any) and gives us Include / Exclude / Converters / Jobs / ConverterMounts. The pre_backup user hook runs; if request.RunJobs=true, the Jobs.PreBackup list runs too.

snapshotVolumes(...) walks the stack’s Docker volumes on the owning node and copies them into staging/volumes/mounts/<vol>/. It respects Include/Exclude globs from the backup config. For volumes with snapshot-capable filesystems (Btrfs, ZFS, LVM — see snapshot.go), backupVolumesWithStop uses a filesystem snapshot to avoid having to stop the stack.

Note: converter-managed mounts are NOT excluded from rsync here. Converters need the data in staging to work on; we delete the raw copy in step 8 after the converter has produced its own output.

Converters are user-declared “dump producers” for e.g. databases: an image that runs pg_dump / mysqldump / mongodump against a volume and emits a logical backup. Each converter:

1. Runs in a container (converterService.Run) against the rsync’d snapshot, not the live data — so no locking required.
2. If successful, its output is copied to staging/converters/<name>/ and a BackupArchiveInfo entry is added to converterArchives with Kind=ARTIFACT.
3. Its raw volume data is deleted from staging/volumes/mounts/ (step 202-207) — the backup stores only the converter’s logical dump, not the raw InnoDB / WiredTiger / etc. files.
4. If the converter fails, the raw data is kept (safer) and the backup continues with a log message.

Six pieces of metadata go into the manifest so restore can reconstruct the stack, not just the data:

WriteManifest and the fullManifest JSON both land in staging/.shc/ so restic snapshots them along with the data.

restic.CreateSnapshot(repo, []string{stagingDir}, tags) creates the actual restic snapshot. On return, createdSnapshotID is set — the orphan guard is now armed.

A BackupCacheEntry row is added to the local cache so the cluster can list backups without roundtripping restic; then the BackupModel row is updated to Status=COMPLETED with size, location (restic:<repo>:<snapshot_id>), and the full BackupManifest JSON.

Instead of calling restic, CreateLocalBackup tars the staging dir into the --output path (optionally compressed per CompressionLevel). The BackupModel.Location records the archive’s filesystem path, and no cache entry is created (local archives are self-describing).

• post_backup user hook.
• If request.RunJobs=true, the Jobs.PostBackup list runs.
• V200401BKPCMP event emitted.
• applyRetentionAfterBackup(tenant, stack) runs retention pruning (see retention module).

The defer block removes the staging directory regardless of success or failure.

If any step from 4-11 returns an error:

1. Forget orphan restic snapshot — if createdSnapshotID was set (step 10a succeeded before a later step failed), restic.ForgetSnapshot is called. Failures here are logged at debug level only — the original error is what the caller will see.
2. Remove orphan cache entry — cache.Remove(backupID) is a no-op when absent, so it’s safe to call unconditionally.
3. Mark BackupModel FAILED — status, ErrorMessage, CompletedAt. A commit failure here is logged and swallowed.
4. Emit V400400BKPFAL event.
5. Return the original error.

Two properties worth noting:

• Orphan safety: if the process crashes between CreateSnapshot and the DB commit, the next backup of this stack will skip the orphan (restic won’t see it tagged with anything useful), but it will still cost repo space until pruned. The explicit forget handles the common case where only the Go side failed.
• Staging cleanup is always attempted: os.RemoveAll(stagingDir) is in a defer block, so it runs in all cases.

RestoreService in restore.go is the inverse, with the twist that restore always targets a possibly-different stack: you can restore backup A into a stack named B to duplicate, rename, or migrate between nodes.

High-level stages (see type documentation for the full API):

1. Extract — pull the backup out of restic (or untar the .bak) into a working directory. The .shc/manifest.json is read first.
2. Plan — compute the target paths, new vault namespace, new Docker project name, new port/host reservations. Conflicts return errors before any data is written.
3. Install the target stack — reuses the install pipeline (see install.md) with variables taken from the manifest. This is what creates the DeploymentModel row, ingress reservations, and runs pre_install.
4. Restore volumes — rsync from the extracted staging dir into the target stack’s volumes. Converter-managed mounts are handled specially: the converter’s restore direction (declared in the converter config) is invoked to apply the logical dump back into the live database.
5. Restore vault — re-import the manifest’s VaultSecrets into the target stack’s vault namespace.
6. Restore integrations — re-create IntegrationModel rows targeting the new stack id (remapped per restore_mappings.go).
7. Start — start the stack via compose up.

Each step is transactional at the step level; a failure at step N leaves steps 1..N-1 in place and raises with enough context to retry or to shc uninstall <target> and start over.

Full reference: errors.md.

• Architecture: backup module
• Architecture: storage module — snapshot capability detection
• Architecture: vault module — how secrets are exported and re-imported
• clone-move.md — uses backup+restore as the transport