Deploy / Restore / Recover — reconstruction model (design)

Status: proposed · Supersedes the piecemeal “republish ports on recover” fix (task #41) with a first-principles model that fixes ports, secrets, integrations, and the unsafe double-boot together.

The Go port kept Python’s recover/restore mechanisms (topo order, rename remap, Capture*, vault export/import) but dropped the thing that feeds them: capturing the effective render-inputs into the manifest and reconstructing from them. Today recover re-renders each stack from the catalog app source with catalog-default vars, so:

• --port host publishes are lost (inline-only, never replayed) → recovered stack publishes nothing. Fixed (proxy-plane v2): --port mappings are persisted in the manifest sidecar and re-injected on every re-render (update / recover / clone).
• @secret! vars (e.g. keycloak adminPassword) re-autogen fresh → password no longer matches the restored DB → kcadm / login fails.
• --integrate edges aren’t in the manifest (inline-only) → recover leans on a group-candidate inference hack instead of recorded edges.

Worse, recover is install (boot against empty data) → restore (swap data in) → force-recreate (boot again). The empty-data boot window is both slow (two boots) and unsafe: the app runs once against empty state — keycloak bootstraps a throwaway admin, and an app wired to a real external service could act on empty local state before the real data lands.

Root cause, one sentence: recover/restore re-derive from catalog defaults instead of reconstructing from the backup, and they boot before the data is in place.

All of install, update, and restore ride the same render → write-compose → compose up engine. They differ on exactly two orthogonal axes:

• deploy unifies install and update into one declarative, idempotent verb (create-if-absent, converge-if-present) — the kubectl apply / terraform apply / compose up pattern. Version is just a field of the spec; “upgrade” is deploy with a different version.
• restore is deploy whose spec comes from a backup, plus a data/vault/volume replay. The data replay is the only thing that distinguishes it — and because converging data is destructive of live drift (it discards anything written since the backup), restore stays a separate, deliberate verb. The render is shared; the data convergence is what’s dangerous.
• recover = restore orchestrated across the dependency graph with a group rename map.

The dividing line is not the render (shared) — it’s whether the operation converges data. deploy never touches data (safe to re-run); restore/recover do (guarded verbs).

Renaming the install/update CLI verbs to deploy is a UX decision, independent of the bug fixes below — the fixes all live in the engine. There is real safety value in keeping restore a distinct, scary word.

Converge a stack to a desired spec (app + version + vars + integrations + ports) from catalog + flags. Idempotent. No data replay. Flags: --integrate, --port, --var, --node, --runtime, version.

Reconstruct one stack from its backup to a target:

• target exists → roll back in place (recreate the shell against restored data).
• target absent / new name → bootstrap (render the packaged app, pre-populate everything, bring up).
• either way: pre-inject vault/vars/volumes/ports before the render+boot.

Rules:

• Every field defaults to the manifest; pass nothing → faithful reproduction (the 99% case).
• Overridable: --integrate plug=provider, --var, --port, etc. override the manifest. Overriding on an existing stack = “restore the data but converge to a modified spec” (restore-and-reconfigure). Generally not done, but it falls out of the model for free and is occasionally useful.
• Version is pinned to the backup. Restore does not change version — that’s deploy. (Restore then deploy if you want a different version.)
• No --rename. Renaming one stack = just name a different target (restore --stack NEW). The stack name is its identity, passed exactly like deploy.
• Integration control is target-dependent: rollback (existing) inherits/reproduces the backup’s edges and usually needs no flags; restore-to-new resolves edges against the target environment, so --integrate (override target), --skip-integration (drop) matter there. --integrate is powerful enough to express any single-stack remapping.

Expand .bkg → topo-sort by manifest integration edges → restore each stack in dependency order → thread a group rename/remap.

• --rename A=A' P=P' is recover-only sugar: it renames the stacks and auto-derives every consumer’s provider remap from the bulk name map (so you don’t hand-write N per-stack --integrate). It expands into per-stack target names + --integrate remaps. Single-stack deploy/restore never need --rename.
• --skip-integration plug drops named edges group-wide.

Because restore bootstraps brand-new stacks, recover no longer does “install then restore” — the old InstallStack+RestoreData two-step collapses into one restore per stack.

Recover/restore reconstruct from the backup, so the backup must capture the effective render-inputs (Python already did; Go dropped most of them):

Note CLI-installed stacks resolve through resolveStackFromDisk (the daemon DB never saw them), which currently populates almost none of these — capture must work in that path too.

Decided (deliberate divergence from Python’s catalog-first): recover/restore render from the packaged pre-templated app in the .bak (exact fidelity — you get back the app that was running), and pin the catalog reference (app name + version) when a matching catalog entry exists, so the recovered stack stays upgradeable (“looks like it came from the catalog”). If the catalog lacks it, the stack is catalog-free but still boots from the packaged source.

Go change: resolveAppDir must prefer the restored <deployDir>/app/ over the catalog on the restore/recover path (today it’s catalog-last-in-list / effectively catalog-only for the value used), and the recovered deployment record must store the catalog linkage.

Decided: pre-populate, not autogen-then-overwrite. Python lets autogen mint a throwaway secret then overwrites it during a late vault import (requiring a re-render + restart). The correct design:

1. Load the backup’s vault secrets into the vault before the render.
2. Autogen is already find-or-create (both Python and Go Get() first) → it finds the restored secret → no mint, correct value in the first render, one boot, no overwrite dance.

Prerequisite: fix Go’s dual-vault split — autogen currently mints into a per-stack JSON store while the exporter reads the DB vault, so the minted secret is neither captured nor where a restore-then-render would look. Mint, export, import, and autogen-lookup must all use the same store.

Intended model (reconstruct everything on the host, then boot once):

lay down packaged app (+ catalog pin)
  → pre-populate vault from backup
  → restore volumes/data (ownership applied, §8)
  → resolve vars (manifest defaults; autogen is now a no-op)
  → render compose (ports injected, integrations wired, mounts chowned)
  → compose up  ← single boot, against correct data

Status: implemented and REVERTED (May 2026). A single-boot reorder (a PreUpHook staging backup data into prepped mounts before one compose up, retiring RecreateOnStart) regressed the keycloak suite 80→76 and ~1.6בd recover runtime (~650s→~1040s, tests 72–75 crash-loop, Connection refused :8080). It was reverted.

Root cause — the second boot is load-bearing for dump-replay apps. Data-as-a-DB apps (postgres, and therefore keycloak’s backing store) do not restore by laying volume files down; they restore via a stop → one-shot restore: job → clean boot choreography: the restore job runs initdb + pg_restore to rebuild the pgdata volume from the dump, then the DB must boot fresh against the rebuilt data. RecreateOnStart’s --force-recreate second boot is that fresh boot. Single-boot races the rebuild → the app connects to a half-initialised DB → crash-loop. (Python is also double-boot here — single-boot was net-new optimisation, not parity.)

Revised design — single-boot is a per-app OPT-IN; the default stays double-boot:

• Default (safe): double-boot via RecreateOnStart, exactly as today. Correct for every app.
• Opt-in: single-boot only for pure volume-file apps — data is just files in a volume, with no DB rebuilt by a restore job. For these the first boot can see correct data directly (the original win: no empty-data window, no throwaway first-run side effects, one boot).
• Detection (proposed, no new per-app field): an app is dump-replay (needs the double-boot) iff its backup.yaml declares a data-rebuilding restore/pre_restore one-shot job; an app with no restore job is pure-volume and may take single-boot. An explicit x-shc-restore: { single_boot: true } marker can force it for edge cases. Default to double-boot whenever in doubt.

Crucially, the keycloak admin-bug fix does not depend on single-boot — it is delivered by rendering-from-manifest + pre-populated vault (legs 1–3, 5a). Single-boot was only ever the perf half of the win, and it only applies to the pure-volume tier.

Question: pre-populating a bind-mount with backup data before the container exists needs the data to have the ownership the container expects.

Finding (evidence-based): SHC already solves this and it is safe for the apps that matter:

• resolveAndPrepVolumes (deploy) and reapplyMountPrepFromCompose (restore) chown the host-bind to the volume spec’s :uid:gid before boot — in-process os.Chown/chmod, falling back to a docker run --user 0 busybox chown -R sidecar. Spec grammar: src:tgt:mode:perm:uid:gid (core/volume/parse.go, prep.go; modules/backup/cli_runner.go for the sidecar).
• The backup tar preserves numeric uid/gid in headers, but Go’s extractTar/mergeDir strip ownership on extraction (data lands daemon-owned) — intentionally; the spec-driven chown re-applies it before boot. (Python instead relied on tar-preserved ownership and the container running as a matching uid.)

App buckets (catalog scan):

• Safe — declare :uid:gid: postgres (26:26), mysql (1001:1001), mongodb (1000:1000), clickhouse (101:101), nextcloud (33:33), matomo (33:33), shlink, wazuh-indexer.
• Safe — init-container chown: mission-control, ciso-assistant, fleet, librechat, frappe, mailcow.
• Risk — no ownership decl, no init container: mariadb (gap: mysql has :0700:1001:1001, mariadb has none), supabase pgdata (a postgres data dir with no :uid:gid — notable), redis, minio, vaultwarden, jellyfin, paperless, immich, gitea, gitlab, keycloak data vol, sonarqube, openobserve, pulp, matrix/synapse. These either run as root (fine) or rely on the image entrypoint to chown (fine if it does so unconditionally) — but it’s unverified per app.

Recommendation (two parts):

1. Add a preserve-on-extract fallback (match Python): when extractTar/mergeDir restore volume data, apply the tar’s numeric hdr.Uid/hdr.Gid when no spec override exists. Result: spec-declared :uid:gid wins (chowned before boot); otherwise the data keeps its original owner. This removes the dependency on the entrypoint for the risk-tier apps.
2. Audit & annotate the risk-tier data dirs that run as a fixed non-root uid — add :mode:uid:gid suffixes (start with mariadb and supabase pgdata). Test restore per app.

Ordered so each leg is independently landable and verifiable. Leg 1 unblocks the rest.

1. Unify vault + pre-populate. Make mint/export/import/lookup share one store; on restore, import backup secrets before render so autogen no-ops. Files: modules/app/install_runner.go (ResolveAutogenInVars/ResolveVaultAutogenInVars), modules/vault/*, modules/backup/orchestrator.go (exportVaultSecrets), modules/backup/restore.go (importVaultSecrets), daemon/backup_seam_adapters.go (SetVaultExporter/SetVaultImporter).
2. Capture render-inputs into the manifest. Add Variables to StackContext + BackupManifest, populate from deployment vars.yaml; ensure ports + integration edges are captured in both the DB-resolver and resolveStackFromDisk paths. Files: modules/backup/seams.go, modules/backup/capture.go, modules/backup/types.go, modules/backup/orchestrator.go, daemon/backup_seam_adapters.go.
3. Restore renders from the manifest inputs. Recover/restore pass manifest.Variables into the render (kills the re-autogen), wire integrations from manifest edges, honor --skip-integration through to render. Files: daemon/services_recovery.go, modules/backup/recover.go, modules/stack/types.go (StackInstallRequest.Variables, .SkipIntegrations).
4. Packaged-app source + catalog pin. resolveAppDir prefers the restored <deployDir>/app/ on the restore path; record catalog linkage on the recovered deployment. Files: modules/app/install_runner.go, deployment record writer.
5. Single-boot reorder + retire RecreateOnStart. ⚠️ REVERTED — do NOT re-attempt as a blanket change (see §7). The double-boot is load-bearing for dump-replay DB apps; single-boot crash-loops them. If pursued, it must be a per-app opt-in (pure-volume apps only), gated on the app declaring no data-rebuilding restore job — and the default must stay double-boot. The reliable-restore value is already delivered by legs 1–4/5a without it.
6. recover = orchestrate restore. Collapse InstallStack+RestoreData into a single restore per stack in topo order; --rename expands to per-stack target + --integrate remaps. This is a code-unification refactor that is independent of single-boot — it must PRESERVE the double-boot (RecreateOnStart) so dump-replay apps keep working. Value is mostly code-clarity, not behavior: recover already renders from the packaged app + manifest and replays data (legs 2/3/5a), so the functional gain over today is small; weigh the refactor risk against it before scheduling. Files: modules/backup/recover.go, daemon/services_recovery.go.
7. Volume ownership. Add tar-preserved-ownership fallback on extract; audit/annotate risk-tier apps (mariadb, supabase pgdata first). Files: modules/backup/archive.go (extractTar/mergeDir), modules/backup/restore.go (reapplyMountPrepFromCompose runs before boot), apps/*.

Old .bak/.bkg lack the new manifest fields (variables, full secrets). Reconstruction must degrade gracefully: missing field → current behavior (re-derive that input). New backups capture the full input set, so a backup→recover round-trip taken after this work is faithful; older archives keep working at today’s fidelity.

• mariadb / supabase pgdata ownership: confirm the runtime uid before annotating (#9 leg 7).
• deploy CLI rename: adopt now or keep install/update? (Independent of the fixes.)
• Restore-to-new integration resolution when a manifest provider is absent and not overridden: error (like deploy’s provider validation) vs warn-and-skip.
• Single-boot opt-in detection (§7): infer the pure-volume tier automatically from “no data-rebuilding restore job in backup.yaml”, or require an explicit x-shc-restore.single_boot marker per app? Auto-inference is zero-touch but must fail safe (default double-boot on any doubt); an explicit marker is auditable but needs every pure-volume app annotated. Whichever — the gain is perf-only on a subset, so this is low priority next to the reliable-restore work that already landed.