Migration recipes

How to give an app a native-migration path from a foreign platform. A migration recipe declares how a foreign app’s dumb backup (crash-consistent volume snapshots that conform to nothing) is transformed into conforming native SHC backup(s) — a backup group the existing restore machinery already consumes. Migration is a data transpile: it introduces no new deploy path. (See the adapter reference for which platforms support migrate, and the glossary for adapter / adapted app.)

Recipes belong to the target native app, under its migrations/ dir:

apps/<target>/migrations/<adapter>.migration.yaml          # canonical
apps/<target>/migrations/<adapter>/<sourceApp>.migration.yaml   # escape hatch

• The adapter is carried by the file name (runtipi.migration.yaml), never by a key in the body — a source.adapter: key does not exist in the schema.
• The canonical single-file shape covers the common case (one recipe per adapter per target). The directory escape hatch exists for a target that needs several recipes for the same adapter (migrations/kube/bitnami-postgresql.migration.yaml, migrations/kube/portainer-postgres.migration.yaml, …).
• A transform job referenced by the recipe (*.job.yaml) sits beside the recipe file and resolves relative to it.

Shipped example: apps/postgres/migrations/runtipi.migration.yaml (+ migrate-linkwarden-pg.job.yaml beside it) — runtipi’s bundled linkwarden postgres transpiled into a native postgres stack.

name: runtipi-linkwarden        # recipe identifier
source:
  app: linkwarden               # foreign app name — string OR list
  # app: [linkwarden, linkwarden-ce]   # one recipe, several store spellings
  match:
    version: '^v?2\.15\.'       # regex on the SOURCE version; capture groups
                                # feed derived target versions (${1}, ${2}, …)
targets:                        # 1..N native apps produced
  - app: postgres               # a bundled-DB foreign app commonly splits
    version: '16'               # into an app target + a db target
    transform:
      job: migrate-linkwarden-pg.job.yaml   # EITHER job: … OR from_volume: …
      produces: files           # exporter | dump | files
      artifact: pgdata          # target app's backup DATA-ITEM to package under
    integrate:                  # optional; the app->db wiring (see below)
      - plug: postgres          # a <plug>.plug.yaml in THIS target's app
        provider: postgres       # another target's `app`
group: consistency              # consistency semantics of the produced group

Field notes (authoritative source: modules/adapter/migration.go):

• source.app — scalar or list; matches the foreign app name the adapter enumerated (for kube, the repo-qualified chart identity).
• source.match.version — a regexp against the source version. The version participates in recipe lookup, so two recipes for the same (adapter, app) with disjoint version ranges are not ambiguous.
• targets[].version — either a capture-substitution template (version: '${1}') or an explicit mapping table for irregular source→target version relationships.
• transform — exactly one driver:
  • job: — a *.job.yaml container run with the source volumes mounted read-only under /shc/source/<volume>; it must never mutate the source. Typical shape: start the engine over a writable copy, take a logical dump, load it into a fresh native datastore, package the result.
  • from_volume: — extract a file tree / logical dump directly from one source volume (no job container).
  • produces: — the output contract: exporter (exporter-format export), dump (logical dump), or files (complete file tree).
  • artifact: — the target app’s backup data-item name the raw output is packaged under (e.g. postgres’ pgdata, so the target’s jobs.restore @pgdata:/backup mount resolves). Empty means the transform already emits a complete .bak and no assembly is needed.
• integrate: — optional per-target list of native integration edges (see Integration wiring below); empty for single-target recipes.
• group — names the consistency group; all targets’ backups assemble into one all-or-nothing restore group (.bkg).

integrate declares the native integration edges a target’s stack must be wired with once the produced group is installed — the migration analogue of shc install -i <plug>=<provider>. A bundled-DB source that splits into an app target + a db target declares the app→db edge here so the restored app talks to the restored db.

Each entry names one of this target’s consumer plugs (a <plug>.plug.yaml in the target app) and the PROVIDER target — another target’s app — whose produced stack the plug binds to.

Two equivalent YAML forms:

# Explicit mapping
integrate:
  - plug: postgres        # this target's postgres.plug.yaml
    provider: postgres     # another target's `app`

# Terse `-i` grammar (identical result)
integrate:
  - postgres=postgres      # <plug>=<provider>

The edge carries no new mechanism. It is resolved into a concrete consumer-plug → provider-stack edge, then stamped into the produced consumer target’s backup manifest as an IntegrationRecord{ConsumerPlugName, ProviderStackName} — the SAME shape a normal backup captures. shc recover already replays each manifest’s captured integration edges (DB-provider-first) when the group restores, so the plug is re-connected exactly as a captured edge would be. ProviderSocketName is left empty, so install re-resolves the socket by plug capability — identical to a bare -i <plug>=<provider>.

The wiring is also surfaced in the plan: MigrationResult.Integrations (the group-wide flattened set) and each consumer PlannedJob.Integrations, so a --dry-run displays every integrate <consumer> -i <plug>=<provider> edge before anything runs.

Each integrate edge is validated at resolve time. A provider that is not a SIBLING target (names the target itself, or an app outside the recipe), or an edge missing its plug/provider, is a recipe-authoring error (X400951ADPMIG), never a silent no-op.

integrate is purely additive. A target that declares none produces a manifest with no integration records — single-target recipes (e.g. linkwarden → postgres) are byte-identical to before.

shc adapter migrate <app> --adapter <name> --version <ver> --from <src> resolves the recipe by globbing both shapes under the target apps tree and body-matching on (app, version):

• no recipe file, a version outside every match.version, or an unmappable version → the run fails with X404950ADPMNF (no migration path for adapter …) — before anything touches the source;
• two or more recipes matching the same (app, version) → a typed ambiguity error (X400951ADPMIG). Disjoint version ranges are fine.

--from accepts three source shapes: ssh://[user@]host (foreign box — quiesced pull via the adapter’s box layout), <tenant>/<name>[@<env>] (a stack SHC already runs, e.g. an adapted app going native — cross-checked against the compat.adapter provenance stamped at load), or a local path (a dumb-backup directory produced earlier).

The engine plans one job per target, runs the transforms with source volumes read-only, assembles the outputs into a conforming consistency group (consistency.bkg), and emits the shc recover command that replays it into the native stack(s). Restoring is the standard restore path — nothing migration-specific survives past the assembled group.

Runtipi ships nextcloud as a compose stack that bundles its own postgres. SHC’s native nextcloud instead consumes a SEPARATE native postgres stack over the postgres plug. The recipe splits the bundled stack into two native stacks AND re-wires them:

name: runtipi-nextcloud
source:
  app: nextcloud
  match:
    version: '^v?(\d+)\.'
targets:
  - app: nextcloud
    version: '${1}'
    transform:
      from_volume: nextcloud       # /var/www/html document root
      produces: files
      artifact: nextcloud-data     # -> the native nextcloud-data volume
    integrate:
      - plug: postgres             # nextcloud's postgres.plug.yaml
        provider: postgres          # -> the postgres target below
  - app: postgres
    version: '16'
    transform:
      job: migrate-nextcloud-pg.job.yaml   # logical pg_dumpall -> fresh cluster
      produces: files
      artifact: pgdata             # -> the native postgres pgdata volume
group: consistency

Restoring the produced consistency group brings up postgres first, then nextcloud with its postgres plug already bound to the restored postgres stack; nextcloud’s post_restore run-upgrade/occ hooks then reconcile schema drift against the restored DB. See apps/nextcloud/migrations/ for the shipped recipe

• transform job.

1. Pick the target app (where the data should land natively) — the recipe lives in that app’s migrations/ dir, named for the adapter.
2. Identify the source’s data volumes (the adapter’s MigrateSources names them — volume names are stable per adapter; recipes’ from_volume: refs lock against them).
3. Write the transform: prefer a logical dump → fresh native datastore job (survives version/layout drift) over raw file copies; mount sources RO.
4. Scope source.match.version to the versions the transform is actually proven against — the regex is the recipe’s honesty contract.
5. For a bundled-DB source that splits into app + db targets, add an integrate: edge on the app target so the restored group re-wires the plug.
6. Test end-to-end with --from <path> against a captured dumb backup before trying a live ssh:// box.