Persist the backup manifest inside the restic repo (2026-07 cross-cluster recover finding)

The cross-cluster no-file migration — backup create on cluster A against a restic repo both clusters can reach, backup sync + backup export + recover on cluster B, with no file ever copied host-to-host — restores the raw DATA correctly but silently drops everything else a recovered stack needs to be reachable: host_mappings (ingress), port_mappings, integration edges, resolved variables, and the stack record. --override host becomes a no-op, recovered stacks lose ingress entirely, and apps with a required plug fail install with X400994PLGREQ. The root cause is a single architectural gap: the RICH manifest is assembled in memory, after the restic snapshot already exists, and is never written into the repo the snapshot lives in — only 4-5 short identity tags are. This proposal decides how the rich manifest travels through the repo so a foreign cluster can reconstruct it without ever touching cluster A’s filesystem.

The gap (all refs verified at current main)

WRITE — CreateBackup (internal/modules/backup/service.go:716) builds the small tag set at :866-880 (backup_id, tenant, stack, environment, and consistency_id when the backup is a group member — 4 or 5 keys, all short strings) and calls restic.CreateSnapshot(ctx, repo, req.Paths, tags, req.Exclude) at :884. CreateSnapshot (internal/modules/backup/restic.go:202) turns tags into repeated --tag key=value CLI arguments (:209-212) — that is the entirety of what rides the restic snapshot object. The RICH BackupManifest — App, AppVersion, Runtime, Variables, PortMappings, HostMappings, Integrations, StackRecord, VaultSecrets, DeploymentConfig — is assembled at service.go:908-991, entirely AFTER CreateSnapshot returns, from a live StackContext (s.resolveStack, :918) that has nothing to do with the snapshot. It is marshalled to JSON at :993 and persisted ONLY as row.Manifest in cluster A’s local backup table (completeBackup, :995-1003). The restic object on disk never sees it.

READ #1 — SyncCache (service.go:1096-1235) is how cluster B learns a snapshot exists at all: it calls restic.ListSnapshots (:1130) and, per snapshot, runs the tags through parseTagMap (:1142, defined :1237-1250) to pull stack/environment/tenant/backup_id/consistency_id into a BackupCacheEntry (:1194-1207). BackupCacheEntry (internal/modules/backup/types.go:827-840) and its ent-backed table (ent/schema/backup_cache.go:26-41) have no manifest column at all — there is nowhere for a richer payload to land even if SyncCache had one to store.

READ #2 — ExportBackupToFile (internal/modules/backup/actions.go:694) is what materializes the LOCAL .bak cluster B’s own recover command reads (recover.go:632-638 hard-requires req.File or req.FromBackup — a real path on disk; LoadBackupManifest, recover.go:312, reads the manifest.json AR member with a plain os.Open, no restic involved). ExportBackupToFile ships row.Manifest verbatim (:730). For a NATIVE row Get resolves this through loadOne against the backup table and gets the rich JSON. For a cluster-B-synced FOREIGN backup, Get (service.go:325-379) finds no backup-table row, falls through to getFromCache (:386-418), which projects the cache row through cacheEntryToModel (:420-460). That function manufactures a BackupManifest from exactly the columns BackupCacheEntry has — ID, Timestamp, Version, TenantName, StackName, Environment, Checksum, Consistency — and nothing else; its own doc comment says the quiet part out loud: “the source cluster’s full manifest lives only in ITS backup table.” This is the “cross-cluster no-file migration” path in its entirety: backup export <id> on cluster B writes a LOCAL .bak sourced live from the shared repo (register.go:960-999, Export) — the data bytes are correct (DumpFile streams the real restic content), but the manifest.json AR member is the eight-field synthesis above.

Consequences, traced to their exact mechanism. applyMappings (restore.go:1165-1229) replays manifest.HostMappings/PortMappings; for the bare synthesis both slices are nil, so the len(manifest.HostMappings) > 0 guard at :1192 is false, the delete-then-insert never runs, and the overrideHostname call that would honor --override host=OLD:NEW (:1221) is never reached — hence “silent no-op.” ComputeRecoverPlan (recover.go:632-820) builds the stack dependency graph by walking l.manifest.Integrations (:732-746); an empty slice means zero edges, so a plug that must reconnect to a provider stack (app/plug_required.go:82, X400994PLGREQ, app/errors.go:1053) never gets wired and install fails loudly. Variables, StackRecord, and VaultSecrets are dropped the same way — the recovered stack re-renders from catalogue defaults instead of the captured render inputs.

One piece of prior art already sits, unused, in the Go port: types.go:787-790 documents ResticBackupManifest as “the manifest written under .shc/manifest.json inside a restic snapshot” — a straight port of a Python convention — and restic.go:490-511 (Restic.CreateManifest) builds the struct in memory. Nothing ever calls it beyond its own declaration; the write-into-the-snapshot half of the port was dropped. This proposal restores that intent with the current, richer BackupManifest shape and closes the loop on the read side, which the Python era apparently never needed to solve in quite this form.

Options: where does the rich manifest ride

(a) Tags only. Cram the marshalled BackupManifest JSON into one large tag value. Rejected: restic tags are transported as individual --tag key=value CLI arguments (restic.go:209-212) and surfaced back as a flat []string per snapshot in restic snapshots --json — every consumer in this codebase, and restic’s own tag UX, treats a tag as a short grep-able label (today’s 4-5 keys are all ≤ ~40 characters). The rich manifest is unbounded — Variables map[string]any, Integrations []IntegrationRecord, HostMappings/PortMappings []map[string]any, and a base64’d VaultSecrets.EncryptedBundle can each run to several KB per stack — and JSON’s =, quotes, and newlines would need escaping through both the daemon’s exec-argv construction (execResticCmd, restic.go:545) and restic’s tag parser. Worse: SyncCache calls ListSnapshots for the WHOLE repo on every backup sync (:1130); if the full manifest rode in a tag, every sync would deserialize every stack’s entire captured state just to read stack/environment — the exact cheap-index property tags exist for would be destroyed.

(b) In-snapshot file only, drop the tags. Write manifest.json as a real backed-up path, stop tagging identity. Rejected: SyncCache’s filtering (tagMap["stack"]/["environment"]/["tenant"], :1147-1167) is O(1) per snapshot from one restic snapshots --json call. Without tags, learning stack/tenant for filtering would require a restic dump PER snapshot — an O(n) restic-process fan-out on every sync, for a table that today does one listing call. That regresses SyncCache from an index scan to a full-repo walk.

Recommendation: (c) hybrid — keep the tags, add one, carry the payload in a file

Leave the existing 4-5 identity tags exactly as they are (backward compatible by construction — old snapshots already have precisely this and nothing more). Add one new tag, manifest_path, whose value is the absolute path, inside this snapshot, where the full manifest JSON lives. The tag stays a short string (a filesystem path, tens of bytes) so the fast-index property of SyncCache’s ListSnapshots scan is untouched; the unbounded payload — Variables, Integrations, HostMappings, PortMappings, StackRecord, VaultSecrets, DeploymentConfig — rides as an ordinary file inside the snapshot, fetched with a targeted restic dump <snapshot> <manifest_path> (the same primitive ExportBackupToFile already uses at :748 and the mount subpackage’s restic source already wraps at restic.go:450) exactly once, lazily, only for snapshots that declare they have one.

Exact convention. Stage the manifest at <SHC_STATE_HOME>/.shc-manifest/<backup_id>.json — a reserved, hidden top-level directory, a sibling of deployments/, never nested under any stack’s deployment path. Pass that path as an EXTRA member of the paths slice on the SAME restic.CreateSnapshot call that captures the app data (restic backup already accepts multiple, unrelated absolute paths in one invocation — nothing new is asked of restic here), so the data and the manifest land in one atomic snapshot with one snapshot ID. Tag that snapshot manifest_path=<SHC_STATE_HOME>/.shc-manifest/<backup_id>.json.

Why a tag-carried path instead of a fixed convention cluster B can compute. The obvious alternative — derive the path purely from backup_id (already tagged) without a second tag — fails across clusters with different SHC_STATE_HOME layouts: cluster B has no way to know cluster A’s state-home root, so it cannot reconstruct the absolute path cluster A actually backed up. Carrying the resolved path AS a tag sidesteps that guessing game entirely — cluster B never needs to know anything about cluster A’s filesystem layout, it just reads the tag and dumps that exact string. This is the one piece of information that genuinely belongs in a tag: not the payload, but the payload’s address.

Sequencing change this forces in CreateBackup. Today the rich fields are built AFTER CreateSnapshot (:908, needing snapshot.ID only for manifest.Checksum). None of resolveStack, CaptureVariables, CaptureHostMappings, CaptureIntegrations, CreateStackRecord, or exportVaultBundle (orchestrator.go:210-230) actually depend on the snapshot existing — they read the live StackContext and the vault exporter, both independent of restic. So the manifest build moves BEFORE CreateSnapshot: resolve the stack, assemble every field except Checksum (left empty — the snapshot ID literally cannot be known before the snapshot is taken), marshal, write to the staging path, append that path to the paths slice passed into ONE CreateSnapshot call. manifest.Checksum is filled in on the LOCAL row.Manifest copy after the snapshot returns, exactly as today; the in-repo copy simply omits it — no consumer needs it, because the reading cluster always trusts snap.ID from restic snapshots --json (the same field SyncCache already uses for ResticSnapshotID, :1196), never a self-reported value inside the file. One observable behavior change falls out of the reordering: a --secrets-password vault-export failure (service.go:960-970) is now caught BEFORE any data is captured, not after — today’s code creates the data snapshot first and then ForgetSnapshots it on a vault failure (:968); under this reordering there is nothing to forget. Strictly an improvement (fewer wasted restic invocations on a failure that was always going to abort the backup), but a real change in the sequence of publisher events an operator might have scripted against.

Clean up the staging file after CreateSnapshot returns (success or failure) — it is now durable inside the repo (or the whole backup failed and nothing needs it). A crash between staging and the snapshot call leaves at most one orphaned few-KB file per crashed attempt (fresh backup_id per call, so it never recurs); not worth more than a comment today, a periodic sweep of .shc-manifest/ is a cheap follow-up if it ever matters.

Restore-path safety: the manifest must never reach a live mount

RestoreSnapshot (restic.go:348-373) wraps `restic restore —target

` and already accepts `includePaths`/`excludePaths` (`RestoreBackup`/`RestoreOpts`, `actions.go:44-52,119`); the DEFAULT recover path calls it with `nil, nil` (`restore.go:1043`) — a full restore. Two independent layers should keep `.shc-manifest/` out of the live app:

Structural. .shc-manifest/<backup_id>.json is a sibling of deployments/, never inside it. replayRestoredDeployment (actions.go:180-) only copies the subtree whose path TAIL matches deployments/<tenant>/<stack>/<env> (:199-227) back onto the live mount — a file living outside that tree is restored into the scratch target (<state_home>/restore/<backup_id>/...) but is never part of what gets copied over the running deployment. This already prevents the worst case (the manifest landing inside a container’s bind mount) as a side effect of the existing replay logic.
Explicit, defense-in-depth. Don’t rely on (1) alone — a caller that passes an explicit TargetPath (rollback, the recover orchestrator) skips the replay step entirely and could restore straight onto a path a container reads. Bake a default exclude into the wrapper itself, not into caller-supplied opts.Exclude: RestoreSnapshot always appends --exclude '*/.shc-manifest/*' (a glob, so it strips the reserved directory regardless of which SHC_STATE_HOME root prefixes it on this particular restore target — the same cross-cluster-portability reasoning as the manifest_path tag) UNLESS the caller is the manifest-read path itself, which uses DumpFile/restic dump <snap> <manifest_path> directly and never goes through RestoreSnapshot at all. The one place that legitimately needs the file reads it by exact path, not through a general restore; every general restore strips it by default.

ExportBackupToFile’s DumpFile(ctx, repo, snapshotID, "/") (:748, used to build the .bak’s data.tar member) does not currently support exclude filters (DumpFile has no filter argument, restic.go:450-465), so the .shc-manifest entry will also appear inside data.tar, duplicating what the .bak’s dedicated manifest.json AR member already carries (writeBakArchive, actions.go:889-918). This is accepted as harmless duplication (a few KB) rather than solved here — filtering dump would need a new restic-wrapper method; not worth it for a redundant copy of data that’s already present in the AR manifest slot.

Cache-side persistence: the ent schema delta

BackupCacheEntry needs a manifest column, mirroring exactly how the primary backup table already does it (ent/schema/backup.go:57: field.Text("manifest").Optional().Nillable()):

// ent/schema/backup_cache.go, Fields()
field.Text("manifest").Optional().Nillable(),

with a hand-written migration in the same style as the two most recent column additions (migrations/20260713090000_deployment-proxied-ports.sql; Atlas auto-diff is not usable in this repo, migrations are hand-written):

-- Add column "manifest" to table: "backup_cache"
ALTER TABLE `backup_cache` ADD COLUMN `manifest` text NULL;

BackupCacheEntry (types.go:827-840) gets a matching Manifest string \json:“manifest,omitempty”`field.SyncCache (:1194-1207) fetches the file at most once per backup_id: when the snapshot's tags carry manifest_pathAND the existing cache row (if any) has no manifest yet, callrestic.DumpFile(ctx, repo, snap.ID, tagMap[“manifest_path”]), store the raw JSON on entry.Manifest. Snapshots are immutable once sealed, so this is a genuine fetch-once-cache-forever — no re-dump on subsequent syncs. When manifest_pathis absent (old-format snapshot) or the dump fails (treat as non-fatal, same best-effort disposition as the existingpublishVaultExportWarning/X200062VLTXSKpattern,orchestrator.go:240-244), leave entry.Manifestempty and fall through to today's behavior. RewirecacheEntryToModel (service.go:420-460): when e.Manifest != "", unmarshal it and use it AS the BackupModel.Manifest, only falling back to the eight-field synthesis when it's empty — that one branch is what makes ExportBackupToFile` ship the real thing for a foreign backup.

CreateBackup’s own local upsertCacheEntry call (:1007-1028) does not need to also populate backup_cache.manifest for correctness: Get (:325-379) always prefers the native backup-table row (loadOne, :336) over the cache projection, so the cache row’s manifest column only matters once it is the ONLY record of a snapshot on a given cluster — exactly SyncCache’s case. Populating it from CreateBackup too would be a harmless symmetry nicety, not a requirement; leave it out to keep the change scoped to the actual gap.

Secrets-inclusion policy

No new secret material is created or exposed by this proposal. exportVaultBundle (orchestrator.go:210-230) already runs unconditionally inside every CreateBackup, today, producing a Fernet-encrypted bundle under either the cluster’s vault master (default) or an explicit --secrets-password (service.go:950-981); that sealed blob already sits in row.Manifest in cluster A’s local DB. This proposal does not touch what gets sealed or how — it only gives the ALREADY-sealed bytes a second storage location (the in-repo manifest file) alongside the existing one (the local DB row). The manifest JSON shape (BackupManifest) is identical in both places; VaultSecrets.EncryptedBundle is []byte ciphertext by construction (types.go:349-365 — its own comment already argues for inline-in-manifest transport specifically because “the bundle is already encrypted… so it is safe to embed in manifest.json”). Plaintext credentials never travel through either the tag or the file; only ciphertext does, exactly as today’s .bak export already ships ciphertext.

The one thing worth being explicit about: a MASTER-sealed bundle (the default, no --secrets-password) is cryptographically useless to a foreign cluster — cluster B’s vault master is a different key. That is not a new problem this proposal introduces (the .bak file-export path has always shipped whatever seal row.Manifest carries, master or password), but making the manifest — sealed secrets included — ride EVERY shared-repo snapshot by default, not just explicit --file exports, does mean more ciphertext permanently at rest in a repo that may be read by operators of multiple clusters. Recommendation: ship this unconditionally (matches the existing unconditional local capture, and the “no backcompat, clean breaks” doctrine — gating it behind a new flag would just be a second flag operators have to remember alongside --secrets-password), but the operator-facing takeaway belongs in the CLI docs for backup sync/recover: --secrets-password at capture time is what makes vault secrets usable on the far side of a cross-cluster migration; without it, secrets are present but permanently inert ciphertext, exactly as today.

Backward compatibility

Old snapshots (pre-this-change) have exactly the 4-5 identity tags and no manifest_path. SyncCache seeing no manifest_path tag skips the dump attempt entirely and produces today’s bare synthesis — zero behavior change, zero errors, matching the manifest format’s existing stated convention of tolerant additive fields with no strict version gate (types.go:479-484, the HostBindSkips doc comment: “the manifest format has no version gate / strict mode, its compat convention is tolerant additive fields”). This proposal follows that same convention rather than introducing a new one.

Migration / rollout

No separate manifest_format version integer is needed. The presence or absence of the manifest_path tag IS the format marker — it already answers the one question that matters (“does this snapshot carry an in-repo manifest file”), and it costs nothing extra to check since SyncCache already parses every tag into a map. An old daemon binary reading a NEW-format snapshot degrades gracefully for the same reason in reverse: parseTagMap (:1237-1250) stores every tag key generically, an old SyncCache simply never looks for manifest_path and ignores it — no crash, no special handling required, today’s behavior exactly. If a future format change is ever needed (a compressed or chunked manifest, say), it gets its own new tag name (e.g. manifest_path_v2) rather than an integer gate, preserving this same tolerant-additive pattern. No flag day, no dual-write period, no reader that needs to understand two schemas at once — this is the cheapest possible rollout precisely because the mechanism was chosen (tag-addressed file) to make it so.

Test strategy

Hermetic. Extend the existing fake-CommandRunner harness (create_sync_test.go’s runnerOutput + newTestServiceWithRestic, and the path-aware dump fixture pattern already proven at actions_test.go:306):

CreateBackup: assert the restic backup invocation’s argv includes the staged .shc-manifest/<backup_id>.json path alongside req.Paths, and that the tag list includes manifest_path=....
SyncCache: a snapshots fixture whose tags include manifest_path, a matching path-keyed dump fixture returning a rich manifest JSON payload (real HostMappings/Integrations/Variables) — assert backup_cache.manifest is populated and a second SyncCache call does NOT re-invoke dump for the same backup_id (fetch-once).
Get/cacheEntryToModel: assert a cache row with a populated manifest column round-trips through ExportBackupToFile with HostMappings, Integrations, and StackRecord intact — this is the regression test for the actual bug (today this path always returns the eight-field synthesis).
Negative/backward-compat: a snapshots fixture with NO manifest_path tag still produces the bare synthesis unchanged — pins today’s behavior for old repos.
Restore-side: RestoreSnapshot’s default --exclude argument is present on every call that isn’t the manifest-read path; a restore against a fake snapshot containing a .shc-manifest/ entry does not extract it into the target directory.

Live acceptance. A real two-cluster proof against a shared restic backend (S3/B2), in the spirit of this program’s existing cross-cluster scratch proofs: cluster A installs a stack with a custom ingress hostname, a required plug wired to a provider stack, and --secrets-password set; backup create against the shared repo; cluster B (no filesystem access to cluster A) runs backup sync then backup export <id> then recover -f <local.bak>. Green = the recovered stack keeps its ingress hostname (or honors --override host), the plug reconnects without X400994PLGREQ, and --secrets-password at capture + the matching --secrets-passwords at recover successfully decrypt the vault bundle. Keep a matching run WITHOUT --secrets-password as the regression twin, asserting the DEGRADED marker surfaces (not a silent gap) and data restores intact.

Effort

Write-side reorder + staging + tag (CreateBackup, restic.CreateSnapshot callers): M — the reorder touches sequencing invariants (vault-export failure now aborts before data capture) more than it touches line count.
Restore-side default exclude (RestoreSnapshot): S — one wrapper method, one constant.
Cache schema + SyncCache fetch-once + cacheEntryToModel rewire: M — a real ent field + migration + the fetch-once bookkeeping, but each piece is small and independently testable.
Test suite (hermetic): M, mostly fixture data; live acceptance is S to script given the cross-cluster proof pattern already exists for other features.
Total: one focused change, no flag day, no dual-schema reader — closer to the shape of the cache-schema-delta proposals already in this program than to a wire-format migration.

Non-goals

No change to the .bak/.bkg AR archive format. manifest.json stays the canonical member name and position; this proposal only changes WHERE the bytes that fill it come from for a foreign backup, not the archive shape itself.
No fix for ImportBackupFromFile’s data.tar re-snapshot asymmetry (actions.go:793-847 snapshots the staged data.tar FILE itself rather than its extracted tree) — a real, separate quirk in the file-transfer path, orthogonal to the shared-repo no-file path this proposal targets.
No restic dump exclude-filter support. The minor data.tar duplication of the manifest file (see Restore-path safety) is accepted, not solved.
No new CLI flag to opt in or out of in-repo manifest capture. Ships unconditionally, matching today’s unconditional local vault-export capture.
No retroactive backfill of old snapshots. Existing repos stay data-only-restorable via the bare synthesis forever unless re-captured; no tool is proposed to walk an old repo and inject manifests after the fact.
No integer manifest-format version field. Superseded by the presence/absence of the manifest_path tag, per Migration/rollout above.
No change to how consistency-group members are tagged or grouped (consistency_id) — each member already gets its own independent snapshot via grouping.go:271-275; this proposal rides that unchanged.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01Q4JcrwAZTK1EsKqD6xP7c3