Plan: hook action lists + runtime refs (full migration, no legacy)

Status: IMPLEMENTED (#79 + phase 8; ref:// + dns:// runtime refs still unbuilt).

Some lifecycle work is more than “run a job after the service is up.” The driving case is keycloak admin self-heal: when the stored admin password has diverged from vault, the only supported, no-internal-schema fix is to take keycloak down, create a temp admin with kc.sh bootstrap-admin (which refuses to run against a live server), bring it back up, then reset the real admin with kcadm — i.e. an ordered sequence that scales the service down and back up around a job, and only on a real mismatch.

SHC already has every primitive for this — the job runner, and the StackController/x-shc-scale scale window the backup module uses to stop a service, do work, and restore its previous replica count. What’s missing is a way for an app’s hooks to express that orchestration. Today a hook is just a list of job-name strings (hooks.post_install: [keycloak-init], internal/modules/app/hooks.go), and a post-up job can’t stop/restart its own service (install_runner.go:623).

This proposal makes a hook an ordered list of actions declared in app.yaml, with a small extensible action vocabulary, a per-run snapshot so actions can be stateful, failure-safe restore, and a ref+hook:// runtime reference convention for dynamic values. It generalizes beyond keycloak: any app can now declare “stop me, do exclusive offline work, start me back to how I was.”

No legacy. The action-list schema is the only hook schema. The old string-list form is removed (not shimmed), and every bundled app’s hooks: block is migrated as part of this change.

A hook phase is an ordered list of actions. Each action is a typed step the runner executes in declaration order, sharing one per-run context.

hooks:
  post_install:
    - job: keycloak-init                 # run a *.job.yaml
    - scale: { keycloak: 0 }             # scale a service (snapshots prior count)
    - job: keycloak-bootstrap-admin
    - scale: { keycloak: "ref+hook://previous" }   # back to the pre-hook count
    - job: keycloak-recover-admin

The vocabulary is open: backup, wait-healthy, exec, conditionals, etc. are future additions. Each new action is a thin wrapper over an existing capability (backup orchestrator, healthcheck poll, …) plus a registry entry — the runner + context are the real investment, the verbs are cheap to add. We ship job/scale/halt now and grow on demand (YAGNI).

When the hook fires, the runner opens a snapshot. The first time a scale action touches a service, the service’s current replica count is captured into the snapshot (this is exactly what the backup EnterScaleWindow does with Normal). ref+hook://previous resolves against that snapshot — the value at hook entry, not “the previous step,” so it’s unambiguous no matter how many scale steps ran.

Restoration is failure-safe, mirroring the backup window’s deferred scale-back: any service captured during a run is auto-restored to its snapshot count when the hook ends, including on a mid-sequence failure, with a loud log if the restore itself fails. So scale: ref+hook://previous is a convenience for restoring mid-sequence (keycloak must come back up before kcadm); the safety net does not depend on the author writing it.

A job step can halt the remaining sequence. The signal is a GitHub-Actions-style marker the job emits on its own line — echo '::shc-hook:halt::' — chosen because that ::command:: form is the recognized convention for a step signaling its runner over captured stdout (GHA workflow commands), so it reads as a convention, not a magic token. The namespace (shc-hook:) makes a collision with real output a non-issue, and the command shape leaves room for future signals (::shc-hook:set name=x::v).

This gives the runner a clean 3-outcome RunJob contract, matched on a whole (trimmed) line, not a substring:

That’s how keycloak stays zero-downtime in the common case: the first job logs in with the vault value; on a match it seeds the realm and emits the halt marker, so the scale-down + bootstrap-admin never run. On a 401 it exits 0 with no marker → the runner continues into the scale/reset steps. On an unreachable server it exits non-zero → abort, no reset.

The marker is the curated halt vocabulary — one constant + doc, not scattered greps.

These are not “magic strings.” By using the ref+<scheme>:// form they are first-class runtime references — the same family as ref+vault:// secret refs (internal/core/config/secrets.go), distinguished by when and who resolves them:

• Syntax: reuse the existing ref+<scheme>://<path> parser.
• Scheme-keyed registry (general, not hook-only): one registry keyed by scheme, each scheme a resolver namespace. The hook runner registers hook; future schemes (node, cluster, dns) register their own resolvers without reworking anything. A resolver gets the path + the surrounding context (which service is being scaled, which node a deploy targets) and returns the value.
• Naming convention (so it stays consistent): ref+<scheme>://<path>, everything lowercase kebab-case, with the entity as the scheme and the attribute as the path — so paths stay short and there’s no nodeid-vs- mesh-ip drift:
  • ref+hook://previous — the snapshot replica count of the scaled service
  • ref+node://ip / ref+node://id / ref+node://mesh-ip — the node in context (e.g. the node a stack is deploying onto)
  • ref+cluster://leader-ip / ref+cluster://node-count Kebab (not node_id) because these are URI paths and a curated SHC vocabulary — URI-idiomatic and uniform. (This differs from ref+vault:// keys, which are snake_case because they mirror operator-defined env/secret names, not a fixed vocabulary.)
• Context-dispatch: the scheme’s resolver uses the surrounding context to pick which entity — previous knows which service, ref+node://ip knows which node — exactly like ref+vault:// keys are scoped by tenant/stack/env.
• Resolution ordering (important): ref+<runtime-scheme>:// values must survive the install-time render + secret-resolution pass untouched and be resolved only at run time. The render/secret resolver must treat unknown (runtime) schemes as opaque (not “unknown provider → error”). This is the one cross-cutting wiring detail to get right.

The first scheme + token we build is ref+hook://previous. node/cluster/ dns schemes are real future candidates (see Extension points) and slot into the same scheme-keyed registry as ~20-line resolvers.

hooks:
  post_install: &reconcile
    - job: keycloak-init                  # UP: login with vault value
                                          #   match  → seed realm, HALT
                                          #   mismatch → continue ↓
    - scale: { keycloak: 0 }              # snapshot taken; keycloak DOWN
    - job: keycloak-bootstrap-admin       # kc.sh bootstrap-admin user tmpadmin (works: server down)
    - scale: { keycloak: "ref+hook://previous" }   # back to pre-hook count
    - job: keycloak-recover-admin         # UP: kcadm as tmpadmin → set admin = vault → delete tmpadmin → seed realm
  post_update: *reconcile
  post_restore: *reconcile

Common case: one cheap login check, halt, done — no scaling, no downtime. Divergence: the only time keycloak briefly scales to zero, using supported tooling, no CREDENTIAL-table writes. This is the “scale-to-0 → CLI → scale-back, only on mismatch” flow, expressed declaratively.

Terminology — two “down” primitives at two granularities:

• scale (service-level): --scale svc=N — brings ONE service’s replicas to 0. The hook scale action uses this.
• stop/start (deployment-level): docker compose stop/start (swarm: scale the whole stack’s services to 0) — halts the whole deployment (StackController.Stop/Start). This is a real internal primitive: restore uses it (stop deployment → replay → start), and it’s exposed as the shc stack stop/start operation.

The keycloak self-heal scales the keycloak service to 0 (service-level) — it does not stop the whole deployment. Both are primitives; neither fires hooks (see Operations vs. primitives).

Hooks fire on operations (user / API-initiated lifecycle actions), each with a pre_ and post_ phase, all declared in app.yaml hooks: under the one action schema. The full set:

Plus the existing post_deployed.

Hooks fire on operations, never on the internal primitives operations are built from:

• Primitives — the scaler (--scale svc=N, service-level) and StackController.Stop/Start (compose stop/start, deployment-level). Called by operations and by the hook scale action. They do not fire hooks.
• The hook scale action drives the scaler primitive directly, so a scale step inside a hook does not re-fire pre/post_scale — no recursion. Only the user-facing shc stack scale operation fires them.

start/stop get no pre/post_start/pre/post_stop phases — but not because they’re “just CLI/HUD.” They’re a real deployment-level primitive that restore uses internally; since hooks fire on operations and never on the internal primitive, restore’s stop/start would not trigger them anyway. We simply don’t see a side-effect use for hooking the operation (the keycloak case uses a service-level scale, not a deployment stop), so we leave the phases out. (restart = stop+start, also no own hook.)

1. Replace the hook schema/parser: hooks.<phase> is []Action, not []string. Remove the string-list code path entirely.
2. Unify backup/restore hooks: move pre/post_backup + pre/post_restore from backup.yaml jobs: + the loadBackupHookJobs loader into app.yaml hooks: under the action schema. Kill the second parser.
3. Add the new phases to AllHooks + wire pre/post_{recover,move,clone,scale, rollback} call sites at their operations.
4. Convert every bundled app’s app.yaml hooks: block to the action form (the common case is a one-liner: - keycloak-init → - job: keycloak-init).
5. No dual-format acceptance, no deprecation window. The action form is the schema.

0. Schema + parser — Action discriminated type (job/scale/halt), replace the []string hook field; reject the old form. Unit tests.

1. Shared scaler seam — lift the scaler primitive (StackScaler / EnterScaleWindow, --scale svc=N) out of internal/modules/backup (or its daemon adapter) into a place the hook runner and backup both call. Hooks get scale-to-N + snapshot/restore. (This is the replica scaler, NOT the StackController compose stop/start primitive — that one stays internal to the restore/start/stop operations.)

2. Hook runner — runAppHookJobs executes actions in order over a per-run snapshot; scale captures/sets; failure-safe auto-restore; halt short-circuits.

3. ref+hook:// resolver — registry + previous; ensure render/secret pass leaves the hook scheme opaque.

4. Keycloak rewire — split keycloak-init into check(halt-on-match) + keycloak-bootstrap-admin + keycloak-recover-admin; wire the sequence. Also teach validateHooks the action shapes (it currently stringifies hook entries, so it must understand {job|scale|halt} maps).

5. Hook-phase coverage — add pre/post_{recover,move,clone,scale,rollback} to AllHooks + wire their call sites; unify pre/post_backup + pre/post_restore out of backup.yaml/loadBackupHookJobs into the action schema (kill the second parser).

   Shipped (#79): The backup/restore UNIFICATION is done — pre/post_backup

   • pre/post_restore fire through the app action runner (app.RunStackHook, wired in the daemon + CLI), every app’s jobs migrated to app.yaml hooks:
   • <name>.job.yaml. Deviation: the restore: file-prep PHASE stays a backup primitive in backup.yaml (it owns the @converter anti-wipe staging coupled to converters:, and is absent from the operations table above) — so loadBackupHookJobs narrows to that one phase rather than being deleted. The five operation phases are in AllHooks + AllowedHookContexts + appHooksDoc (declared, validated, parseable) AND their firing is now wired at every operation, via app.RunStackOperationHook (self-building deployed-stack ctx over the shared runAppHookJobs):
   • scale — Service.Scale fires pre_scale before the scaler, post_scale after the replica write (success-only). In-module. The scale: hook ACTION drives the scaler primitive directly and does NOT re-fire (no recursion).
   • rollback — rollbackFailedInstall fires pre_rollback after its install-only gate, post_rollback BEFORE artifact removal. In-module.
   • clone / move — fired from the clone module through a new clone.SetStackHookFirer seam (B3: clone can’t import app), wired in the daemon + CLI. pre on the source, post on the recovered/renamed target; fast-path rename fires pre/post_move only.
   • recover — fired from the backup module: the sync RecoverFromFile path via fireStackHookPhase, the async orchestrator via the daemon’s RecoveryActions adapter (app.RunStackOperationHook). Once per recovered stack; wraps the inner install + restore hooks. pre/post wrap the outer operation; inner install/uninstall/restore hooks keep firing as before (verified no double-fire). pre_recover/pre_rollback no-op when the stack isn’t deployed yet (http/container contexts only).

6. Migrate all apps — convert every hooks: block (incl. the backup.yaml ones moved in phase 5).

7. Verify — unit + full e2e; a keycloak divergence test if feasible on the harness.

• Actions: backup (wrap the backup orchestrator), wait-healthy, exec, if/conditional steps.

• Runtime ref schemes (the scheme-keyed registry pays off here):

  • ref+node://ip / ref+node://mesh-ip / ref+node://id — the node in context, e.g. the IP of the node a stack is being deployed onto — a genuinely useful contextual value an app can template without knowing it at render time.
  • ref+cluster://leader-ip / ref+cluster://node-count — cluster-wide live facts (node count → “one replica per node” scale targets).
  • ref+dns://… — DNS already has a bespoke runtime resolver (dns.Service.ResolveTarget, service.go:63) that classifies a --host target into live node IPs / advertise address / LB target. It could fold under this convention later (cosmetic, not a need).

  These are real-but-later: most config is fine at render time (a redeploy re-renders). Runtime refs earn their keep only for values that track live state without a redeploy or are needed mid-operation. The scheme-keyed registry (phase 3) keeps each new scheme a small isolated resolver — so we build hook now and add node/cluster when something needs them, not speculatively.

  Shipped (phase 8): ref+node://{ip,id,mesh-ip,internal-ip,hostname} and ref+cluster://{leader-ip,node-count} are now registered resolvers (runtime_refs.go). The daemon pre-resolves them off the node repository + cluster LeaderInfo and passes them down ComposeRunOptions.{NodeRefs, ClusterRefs}; the app module does pure map lookups (no DB/node/cluster import for the resolver itself). They resolve in two positions: a scale: target (e.g. scale: { app: ref+cluster://node-count }) and — newly — a hook job’s environment: value (a value that is a whole ref+…://… resolves through the registry; sibling jinja values render as before). A known key with no live value (daemonless CLI install, election in progress) resolves to ""; an unknown key (author typo) fails the hook loudly. ref+dns://… remains unbuilt.