Integration teardown: wake the 28 dead jobs, then collapse three runners into one

Revision 2. Rewritten after an adversarial review returned ship with changes. What moved: the dead-job count is remeasured across both catalog trees (26 → 28); the gating mechanism is now the core/lifetime scope parameter, not an operator flag and not an app-authored cleanup@boolean; the rollout reflects the renames that have already landed; and the project owner’s vApp constraint is recorded as a hard constraint rather than an open question.

Twenty-eight job blocks in the catalog were written by app authors, are parsed by nothing, and have never executed. They are not stale leftovers — they are the teardown halves of integrations that ship today.

The catalog lives in two trees. ce/builtin/<app> is the source of truth for the four built-in apps (keycloak, openobserve, postgres, traefik); it is gitignored in this repo and staged into apps/ at build time. The first scan walked apps/ only and therefore missed it. Counts below are a YAML parse of every jobs.<phase> list in the CE repo’s builtin/ and this repo’s apps/.

The apps column reproduces the original scan exactly (42 / 28 / 21 / 10, and 15 + 11 dead) — the two tables differ only in that the dead plug/socket phases are now spelled unintegration rather than cleanup, because the rename in rollout step 1 has already been applied to the working tree. The correction to the total is entirely the ce/builtin column: 2 of the 28 dead jobs live in the staged tree —

• ce/builtin/keycloak/sso.socket.yaml :: delete-oidc-client (unintegration)
• ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions (post_deployment)

The 15 dead plug jobs span 10 apps (defectdojo, fleet, gitea, immich×2, mailcow×2, mautic×2, nextcloud, sonarqube, windmill×2, wordpress×2). The 12 dead socket jobs span 12 apps (crowdsec, hockeypuck, immich, keycloak, minio, paperless, peertube, penpot, pmm, postiz, s3, wazuh).

Lesson for the audit itself. Any catalog measurement that reads apps/ without staging ce/builtin is wrong by construction, and will stay wrong silently because the missing apps are the platform apps — the ones every other app plugs into. Every future catalog assertion must run against the staged tree.

The runners parse, exhaustively:

• plugJobsDoc (plug_jobs.go:43-47) — post_deployment ONLY.
• SocketJobsDoc (socket_jobs.go:45-50) — discovery, integration.
• vsocketDoc (vapp_runner.go:495-507) — discovery, integration, cleanup.

So cleanup executes on a vsocket and is silently ignored on a socket and a plug, and post_deployment is the mirror image — it executes on a plug and is silently ignored on a vsocket. The same word, in the same YAML dialect, in the same directory, with opposite behaviour, in both directions.

plug_jobs.go:41-42 tells the reader the other phases are “handled elsewhere”; they are not handled anywhere. integration_service.go:176-179 is honest about it — “the unintegration + cleanup job phases are out of scope here” — and IntegrationService.Disconnect (integration_service.go:180) has zero callers in the tree.

None of this fails loudly because yaml.Unmarshal into a typed struct discards unknown keys. An author writes a phase, the parser drops it, nothing logs.

• Uninstalling a consumer never revokes its provider-side access. Ten apps wrote SSO/SMTP unwiring (mailcow, nextcloud, gitea, wordpress, …) that has never run once. A removed Nextcloud leaves its Keycloak client, its SMTP relay credential and its bucket IAM user live.
• The only teardown that DOES run is the vApp path, and it runs unconditionally — UninstallVapp loops vsocket.Jobs.Cleanup with no gate of any kind (vapp_runner.go:437-445). Ten of the twelve vsocket cleanups destroy data: postgres/mariadb/mysql DROP DATABASE + DROP USER, mongodb dropDatabase, minio/s3 bucket rb --force, keycloak DELETE /realms/<realm>, redis SCAN+DEL over the whole namespace, frappe and nginx rm -rf. (The remaining two — wordpress and nextcloud uninstall-plugin — remove an app, not a dataset.)
• The migration case that prompted the audit: a bucket holding 55.6 GiB of replicated Nextcloud data, adopted rather than provisioned, destroyed on teardown with no opt-out.

Of the 28 dead jobs, exactly two destroy data on the edge surface — apps/minio/s3.socket.yaml and apps/s3/s3.socket.yaml, both delete-s3-bucket, both rb --force. The other 26 release access, deregister an agent, unwire a config file, or — in the lone vsocket post_deployment case — install postgres extensions. That asymmetry is what makes waking the phases tractable: the blast radius of “parse unintegration” is two jobs wide, and both are already identified.

This is the single most important fact in the proposal, and it is why the dead phases cannot be fixed by adding a struct field.

VappState persists the entire rendered context — PlugData, VVars, Discovery, Preset/PresetResolvedVars (vapp_runner.go:70-110) — with the reason stated inline at :91-94:

“PlugData / VVars are persisted so the cleanup jobs can render against the same context the install used. Without this the {{ integration.plug.data.domain }} ref in the uninstall job has nothing to bind against and the cleanup runs blank.”

PersistedIntegration (install_runner.go:2043-2064) is five bare strings: Plug, ProviderStack, ProviderTenant, ProviderEnvironment, ProviderConnection. No plug data, no socket data, nothing rendered.

The vApp surface persists its bind-time context and its teardown phase works. The integration-edge surface does not persist its bind-time context and its teardown phase does not exist. That is not two independent gaps; it is one gap and its consequence. A teardown job that says mc rb s3/$${PLUG_BUCKET} has nothing to render PLUG_BUCKET against.

Parsing the phase without fixing persistence is worse than leaving it dead: the jobs would run with blank variables. mc rb s3/ and DROP DATABASE IF EXISTS "" are not no-ops in every implementation, and rm -rf "$${DOC_ROOT}" with an empty DOC_ROOT is a catastrophe.

The project owner’s constraint, recorded verbatim:

vApps destroy by default, with no opt-out.

This is a hard constraint on the design, not a default to be revisited. Its consequences run through everything below:

• The core/lifetime parameter described in §2 governs integration edges only. runVappJob never consults it.
• A lifetime: key on a *.vsocket.yaml job is a validation error, not an ignored field. There must be no syntax that looks like a vApp opt-out.
• The ten destructive vsocket cleanups stay unguarded, and that is now the documented intent rather than an oversight. Revision 1 proposed guarding them; that proposal is withdrawn.
• Adoption of a pre-existing resource is therefore an edge operation, never a vApp. A vApp instance is its resource: postgres/database means “this database exists because this vApp exists”. If an operator needs to point at a bucket or database they already own and keep, that is -i <plug>=<provider> with lifetime: external, or -i <plug>=@<connection>. The 55.6 GiB bucket belongs on the edge path, and the fix for that incident is to make the edge path exist and be safe — not to weaken the vApp contract.

They are genuinely different and the distinction is worth preserving:

• cleanup — this thing is being uninstalled. The resource is going away. Fires for a vApp instance and for an app.
• unintegration — an edge is breaking, both endpoints survive. Fires on disconnect and on consumer uninstall.

Under that rule the catalog was misnamed in one direction only: the socket cleanup blocks (remove-bouncer, remove-agent, six cleanup-access) and the 15 plug cleanup blocks all fire when a CONSUMER goes away — edge teardown. They become unintegration. The 12 vsocket cleanup blocks are correct as-is. This rename has landed — see the rollout below.

The remaining misname is the one the recount exposed: post_deployment on a vsocket. ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions means “after the vApp’s database exists”; post_deployment on a plug means “after the CONSUMER’s containers are up and healthy”. Same word, different event. The job is a second step of provisioning and belongs in jobs.integration after vapp-create-database, which already runs in declaration order. One file, one move.

And the parser must stop swallowing this class of bug. yaml.Unmarshal into a typed struct silently drops unknown keys, which is how 28 jobs got authored into the void. The fix is not Decoder.KnownFields(true) — the job docs are partial views of files that also carry schema:, data:, services:, so strict decoding would reject valid catalog. The fix is to decode jobs into map[string]yaml.Node first and hard-error on any key outside the surface’s known phase set. Cheap, exhaustive, and it makes “authored but never executed” unrepresentable rather than merely tested-for.

Destruction is gated by one platform-owned parameter, lifetime, resolved through the existing SHC scope ladder in core/scope (core/scope/scope.go:24-38): Deployment → Stack → Environment → Tenant → Global. Hence core/lifetime: the core/ namespace marks it as platform-defined — the platform owns its name, its value domain, its default and its validation, so no app can misspell it and no app has to re-implement its truthiness.

Value domain — a closed enum, validated at parse time:

Resolution, once, at bind time:

1. explicit --lifetime managed on the install / connect invocation;
2. core.lifetime through the scope cascade — shc config set core.lifetime managed --tenant dev makes a scratch tenant reclaim everything, while production keeps the global external;
3. the plug’s declared default in <plug>.plug.yaml;
4. external.

The resolved value is persisted on the edge (§3) and that persisted value is what teardown reads. Resolving at bind and freezing it is the same argument as persisting plug data: the disposition of a resource is a fact about how it came into existence, and a config change six months later must not silently convert “retain” into “destroy”.

How a job declares itself destructive. The author marks the job; the platform decides whether it runs:

jobs:
  unintegration:
    # Always runs: release the per-consumer credential.
    - name: revoke-access
      context: container
      command: |
        mc admin policy detach s3 "policy-$${PLUG_BUCKET}" --user "$${PLUG_ACCESS_KEY}" || true
        mc admin user remove s3 "$${PLUG_ACCESS_KEY}" || true

    # Runs only when the edge resolved to lifetime: managed.
    - name: delete-bucket
      lifetime: managed
      context: container
      command: mc rb s3/$${PLUG_BUCKET} --force

Two properties follow that the previous designs did not have:

• Release-access and destroy-resource stop being the same job. Today delete-s3-bucket does both in one command, so any guard that skips the destroy also skips the credential revocation — the exact leak the whole proposal exists to close. Splitting them is only possible because the gate is per-job.
• Shell truthiness leaves the blast radius. The runner compares an enum it parsed; no catalog job ever tests a boolean in sh again. This repo has been bitten by that class twice in the last week alone: the jinja fix at core/jinja/jinja.go:865-887 (a stringified "False" was truthy, so huly, affine and gitlab addressed a plaintext S3 endpoint over TLS) and its shell mirror, b9c50518 “a YAML bool never equals the string true in a shell test”. The in-flight cleanup@boolean guards already had to write case "$${PLUG_CLEANUP}" in true|True|TRUE|1) to survive it. A destructive guard is the last place that should depend on which of five spellings of true a template happened to emit.

Default is retain, and the failure modes are the argument. An orphaned bucket costs pennies and is swept later; a destroyed one costs whatever was in it. An edge whose persisted state predates this design, or was lost to a partial restore, resolves to external — so an unknown edge behaves like one that declined. That is what makes waking these phases safe rather than merely possible.

A KeepIntegrations operator flag was prototyped and has been reverted. It is worth being precise about why. The first reason is the one that would have shipped a data-loss bug; the other three apply to any flag-shaped fix, so a narrower flag in a better place would still have been the wrong answer.

1. It was structurally in the wrong place. The gate sat inside ExecuteComposeDown, below the vApp early-return at install_runner.go:1440-1442:

   if (strings.Contains(opts.AppName, "/") && !IsExplicitInstallSource(opts.AppName)) ||
       strings.Contains(opts.StackName, "/") {
       return UninstallVapp(ctx, opts)
   }

   Every vApp uninstall returns there. The vApp path is the only path in the product that currently runs destructive teardown (vapp_runner.go:437-445). So the flag guarded exactly the code paths that had nothing to guard, and never once fired on the path that did — while reading, in the CLI help and in review, as though it protected everything. A flag that is silently inert on its most important case is worse than no flag: it converts an unguarded destroy into an unguarded destroy that someone has already signed off on.

2. Per-invocation disposition is the wrong lifetime. A flag is a property of one command. Whether a bucket survives then depends on how the last uninstall happened to be typed. Worse, several teardown paths have no human at the keyboard at all: the lease reaper (crons_lease.go:199 → DefaultLeaseReapTrigger at :348), the clone/rollback seams (daemon/clone_adapters.go:525, 547, daemon/services_seams_clone.go:389, 404) and the post-teardown PurgeStackRow tail all reach uninstall with a synthesised request. A flag they cannot set is a flag that defaults them into whichever behaviour the code happened to pick — and the reaper, which removes deployments nobody is watching, is precisely the caller you least want guessing.

3. It asked the wrong actor at the wrong time. The platform does not know whether a resource was created or adopted. Only the consumer’s operator knows, and they knew it at bind time — possibly months earlier, possibly a different person. Asking at teardown time is asking someone to recall a fact nobody wrote down.

4. It left no trace. Nothing was persisted, so nothing could be audited, replayed, reported by shc ls, or reasoned about before running the uninstall. There was no way to answer “what happens if I remove this?” other than removing it.

core/lifetime inverts all four: it is resolved at bind (2, 3), by the party who knows (3), persisted on the edge (4), and consumed by the job runner rather than by the uninstall entry point — so there is no early-return it can sit below (1). Because it is a scope parameter rather than a flag, it also gains the tier resolution a flag structurally cannot have: an estate-wide posture, a per-tenant override for scratch environments, and a per-edge exception, all in one key.

Revision 1 proposed a per-plug cleanup@boolean schema field guarded in each job’s shell. It is strictly better than the flag and it is what the in-flight guards on apps/minio/s3.socket.yaml and apps/s3/s3.socket.yaml currently implement. It is still the wrong long-term mechanism:

• It is re-implemented per app, in shell, against jinja’s stringified booleans — see the truthiness argument above. Every copy of that five-way case statement is another chance to write it wrong once, in the destroy path, in a job nobody runs until the day it matters.
• It has no scope. There is no way to express “this whole tenant is scratch” or “this estate never destroys” without editing every plug.
• It is invisible to the platform: shc cannot report it, validate it, or refuse a destroy, because it is just another opaque string in plug.data.
• It conflates the two halves of teardown, as shown above.

The in-flight cleanup@boolean guards are not wasted work — they are the correct interim state (see rollout step 2), and they mark exactly which jobs get lifetime: managed when the parameter lands.

Give integration edges the same bind-time replay VappState already gives vApps — but not by symmetry alone. The two halves of the rendered context need opposite treatment, and the reason is a security boundary the vApp case does not have.

Plug data is replayed. Re-rendering it at teardown reads the consumer’s deployed app/ tree and its vars as they are at uninstall, not as they were at bind. If vars.postgres.database drifted between the two, the teardown job targets a different database than the one provisioned — it does not fail, it succeeds against the wrong resource. Bucket names, database names and usernames must come from the bind-time snapshot. This is the same contract port mappings and var overrides already have.

Socket data is re-rendered, deliberately. The provider’s admin credential is exactly the field you want current at teardown — a rotated superuser password must authenticate, and a replayed one will not. And an edge can be cross-tenant, where integration_render.go:507-520 already blanks every @secret socket field in the consumer-facing copy. Persisting provider secrets into the consumer’s state.json would hand the consumer’s tenant the very values that redaction exists to withhold. The provider is by definition still running when a consumer is being removed — if it is not, the teardown job could not execute anyway — so re-rendering from the live provider is both safe and required.

Where it goes. Not in PersistedIntegration. That struct is one of three that are converted positionally — IntegrationSpec (integration_render.go:50-72), PersistedIntegration (install_runner.go:2043-2064) and DiskIntegration (disk_lookup.go:297-310), each with a comment insisting ProviderConnection stay last because IntegrationSpec(pi) / PersistedIntegration(spec) / DiskIntegration(in) are direct conversions. Adding a map to one silently mis-maps the others, and IntegrationSpec is the type -i parses into — it has no business carrying a rendered blob.

Instead, add a sibling record modelled on VappState, stored as its own key in the deployment’s state.json (JSON, keyed by name, no positional coupling):

// PersistedEdgeState is to an integration edge what VappState is to a
// vapp instance: the bind-time context, replayed at teardown.
type PersistedEdgeState struct {
    Plug     string         `json:"plug"`
    Lifetime string         `json:"lifetime"` // resolved core/lifetime, frozen at bind
    PlugData map[string]any `json:"plug_data"`
}

The three positional structs are untouched. PersistedEdgeState inherits the same at-rest posture the vApp state file already has for the same reason (plug data carries consumer-side secrets such as clientSecret and the vApp DB password); nothing new is exposed, and the provider’s secrets are never written into it at all.

An edge with no PersistedEdgeState — pre-existing deployments, or a state file lost to a partial restore — renders integration.plug.data from the consumer’s deployed app/ tree as a fallback and is forced to lifetime: external. This is not a compatibility shim; it is the permanent rule for any edge whose bind-time context is unavailable. Unknown provenance never destroys.

runVappJob (vapp_runner.go:586-709), runOneSocketJob (socket_jobs.go:402-500) and runOnePlugJob (plug_jobs.go:143-211) are three retypings of one skeleton: default context → validate → render → $$→$ unescape → renderHookEnv → dispatch. The $$ unescape alone is duplicated verbatim at vapp_runner.go:668, socket_jobs.go:423, plug_jobs.go:164.

They converge only in the last four inches — ExecuteResolvedJob, renderHookEnv, ResolveVaultAutogenInVars, LoadVars, RenderMapStrings. There is no shared “build a job template context” function, and the drift shows:

• Different defaults for an omitted context: — vapp service (vapp_runner.go:597-600), socket container (socket_jobs.go:409-411), plug service (plug_jobs.go:150-153). Unifying naively changes the meaning of every catalog job that omits the key.
• Robustness on one side only. Socket-only: context: http probes (socket_jobs.go:360), max_retries / retry_interval (socket_jobs.go:81-87, 298-301), dockerNetworkAvailable fail-fast with X500963NETUNR (socket_jobs.go:473-474), SocketServiceHost fallback (socket_jobs.go:484), mint-and-return via ::shc-connection-value:: (socket_jobs.go:239), @secret redaction on cross-tenant edges (integration_render.go:507-520). The vapp runner rejects http outright (vapp_runner.go:602) and has no retry concept at all — which is why a vApp teardown against a half-stopped parent just loses.
• discovery means opposite things. Socket discovery is a readiness probe whose stdout is DISCARDED (socket_jobs.go:307). Vapp discovery is value-producing: stdout is parsed, published as {{ discovery.<job> }}, and it triggers a SECOND render of the vplug data: block (vapp_runner.go:302, 329).
• pickJobContainerNetwork is defined in vapp_runner.go:730 and borrowed by both the socket runner (socket_jobs.go:458) and the render path (integration_render.go:814) — while socket_jobs.go:10 claims it “mirrors vapp_runner.go::runVappJob”. The dependency runs backwards from the documentation, and the helper hardcodes DockerProjectName("default", …) (vapp_runner.go:757) — a single-tenant vApp assumption sitting in the cross-tenant path, with the comment at :752-756 admitting it.

Target: one job runner and one context builder, parameterised by lifecycle event and provider identity. cleanup vs unintegration becomes two values, not two code paths — and lifetime is checked in exactly one place.

vars resolves four ways, two of them ninety lines apart in one file:

So inside ONE .vsocket.yaml, {{ vars.password }} is the parent superuser in data: and the per-vapp user in jobs.*[].command. This inversion is deliberate and load-bearing. ce/builtin/postgres/database.vsocket.yaml:17 publishes adminPassword: "{{ vars.password }}", and ce/builtin/postgres/database.vvars.yaml declares its own password: ref+vault://vapp_db_password?autogen — the per-vapp DB user’s secret. The comment at vapp_runner.go:534-541 spells out why parent must win there: a vapp’s vvars password must not shadow the parent superuser password the socket publishes. Invert it and every vApp provisions its database with the wrong credential — succeeding, silently, until something tries to log in.

Three consequences, all of which bind work already in flight:

1. A shared context builder that picks one merge order silently mis-provisions credentials. Any unification must keep merge order a per-surface property, with a table test pinning all four before a line of the collapse is written.
2. The parent-wins order is pinned in two places — the install render (vapp_runner.go:264) and the teardown replay (vapp_runner.go:429). Both call SocketVars. A refactor that fixes one and not the other makes install and uninstall disagree about which database they are talking about.
3. The in-flight vars. → vvars. rename sweep (17 refs) must not be applied blanket. In postgres and s3, vars.username / vars.password inside a vsocket data: block mean the parent’s admin credentials by design; renaming them to vvars. swaps admin for instance and inverts exactly the case the comment warns about.

Ordered so each step is separately revertable and none silently destroys data. Status reflects the working trees as of this revision.

Steps 3–6 are small and independently useful. Step 7 is the real refactor and should not be bundled with them.

Ordering constraints that are not negotiable:

• 5 before 6. No phase is parsed until the gate that governs it exists.
• 4 before 6. No teardown job runs until there is a bind-time context to render it against, and a rule for what happens when there isn’t.
• 7 last. Collapsing runners while the surfaces still disagree about vars is how the credential inversion ships.

• Parser exhaustiveness, as an error not a test. Every jobs.<phase> key outside a surface’s known set hard-fails at load. The accompanying test asserts the catalog parses clean — run against the staged tree, so ce/builtin is included by construction.
• Table test pinning the four vars merge orders, including the vsocket data: vs job-command inversion, with the postgres/database admin-vs-instance password case named explicitly, and both SocketVars call sites (install render and teardown replay) covered.
• Lifetime resolution: each tier of the cascade wins over the next; the --lifetime flag beats the cascade; the plug default beats the built-in; absent everything yields external; an unknown value is a parse error, not a fallback.
• Lifetime freezing: change core.lifetime after bind, tear down, assert the bind-time value governed.
• vApp constraint: a vsocket job carrying lifetime: fails validation; a vApp uninstall runs its destructive cleanup with core.lifetime=external set at every tier.
• Teardown behaviour: runs on consumer uninstall while containers are up; releases access unconditionally; destroys only at lifetime: managed; continues past a failing job; forces external and says so when PersistedEdgeState is absent.
• Round-trip: install → uninstall with lifetime: managed destroys; with external the resource survives and a reinstall re-adopts it, with the same bucket/database name the first bind used.
• Disconnect gets its first test alongside its first caller.

Parse unintegration only, leave the vsocket post_deployment dead. Fixes 27 of 28 and leaves one word still meaning two things on two surfaces. Rejected — the one remaining case is a single catalog line.

Make cleanup the single name everywhere. Fewer renames (12 vsocket files already use it), but it collapses two genuinely different lifecycle events — “I am being removed” and “an edge to me broke” — into one word, and a provider must distinguish them: remove-bouncer unwires one consumer, delete-realm destroys everything.

KeepIntegrations operator flag. Prototyped and reverted. Four reasons, detailed above; the decisive one is that its gate sat below the vApp early-return at install_runner.go:1440-1442, so it never fired on the only path that currently destroys anything.

App-authored cleanup@boolean in plug data. Better than the flag, and the correct interim state, but it re-implements a destroy gate in shell once per destructive job against jinja’s stringified booleans, has no scope, is invisible to the platform, and cannot split release-access from destroy-resource. Superseded by core/lifetime; the in-flight guards convert.

Guard the vsocket destroyers too. Revision 1 proposed this. Withdrawn — it contradicts the owner’s constraint that vApps destroy by default with no opt-out.

Re-render plug data at teardown instead of persisting it. Cheaper, and it is what a narrow fix does, but it binds teardown to vars as they are at uninstall rather than at bind, and silently targets the wrong resource after var drift. Retained only as the forced-external fallback for edges with no persisted state.

Persist socket data alongside plug data, mirroring VappState wholesale. Rejected on two counts: a rotated provider credential must be current at teardown or the job cannot authenticate, and on a cross-tenant edge it would write the provider secrets that integration_render.go:507-520 deliberately redacts into the consumer’s own state file.

pre_uninstall / post_uninstall app hooks. Both are fully wired (types.go:97-98, hooks.go:29-30, install_runner.go:1471, 1550, service.go:654, 659) and used by zero apps in either tree. Revision 1 proposed folding them into an app-level cleanup. Deferred: they carry a real before/after-teardown ordering distinction that cleanup does not, they are not part of the integration surface, and folding two working-but-unused hooks is churn that would enlarge an already-large change. Revisit once the first app needs one.