Skip to content
SHC Docs

Lifecycle hooks

Layer 11. Lifecycle. Lifecycle

Hooks let an app run jobs at well-defined points in the SHC orchestration pipeline, e.g. initialize a database after pre_install, dump it before a backup, notify an external system after install completes.

Source of truth: modules/app/hooks.go (the HookExecutor), modules/job/types.go (the JobDefinition schema), modules/job/service.go (JobLoader, JobExecutor).

For where exactly each hook fires in the pipeline, see the flow docs: install · upgrade · uninstall · backup/restore.


HookFires at
pre_installAfter variables are merged, before compose up (install flow)
post_installAfter every replica of the deployment reads ready (Readiness comes first), before the A200100STKINS deployment-installed event
pre_updateBefore re-rendering compose during update
post_updateAfter every replica of the deployment runs the new config and reads ready (Readiness comes first)
pre_uninstallBefore compose down (uninstall step 6)
post_uninstallAfter compose down and network removal
pre_backupBefore volumes are rsync’d into staging
post_backupAfter restic snapshot completes
pre_restoreBefore extracting the backup into target deployment volumes
post_restoreAfter volumes are restored and converters applied, with the deployment running. On a clone or move target that is once the target is brought up, and a failure there fails the clone or move

All hook names are checked against hookNames in hooks.go; unknown names are ignored.

post_install does not fire when the containers have started. It fires once every replica of the deployment reads ready: each service is running and, where it has a healthcheck, healthy, and a restart “no” service has ended 0. An install whose replica never reads ready ends X503764PODWAI, is rolled back, and never runs its post_install hook.

So a service’s readiness must never wait on something a post_install job delivers. The hook waits on the service and the service waits on the hook, and the install rolls back at the 15-minute readiness limit. A start_period does not help: a container inside its start period reads starting, which is not ready.

post_update keeps the same contract. An update’s replicas reach their nodes through the rollout once the update’s spec is committed, so an update with a post_update chain commits, waits until every replica runs the new config and reads ready, and fires the chain only then. Its jobs therefore run in the new containers, wherever they are placed. An update whose replicas do not all read ready within the same 15 minutes never runs the chain and fails with X500908HKFAIL. An lxc start the node refuses counts as refused at every start only once the node refuses it again, so a start that fails once and starts on the node’s retry within the window does not fail the update. An update that fails its readiness gate keeps its post_update chain pending, which the deployment’s next shc update (its plan names the chain) or shc start runs. An update with no post_update chain returns without waiting for the rollout.

shc install --no-wait does not skip this wait for an app with a post_install chain: the chain still fires only once every replica reads ready, and the install still ends X503764PODWAI when one never does. For an app with no post_install chain, --no-wait returns once the replicas are handed to their nodes. A replica that fails to start afterwards is not rolled back; shc status shows it. A replica the install’s own node could not start still fails the install at once.

A service that needs something only the hook can produce (a binary from an app store, a minted key) keeps waiting for it instead of exiting, and its healthcheck passes while it waits:

  • Nextcloud’s notify-push waits for the binary nextcloud-init installs. Its probe passes while the binary is absent and is the listener’s answer once it exists (apps/flagship/nextcloud/compose.yaml).
  • mailbox-reconciler reads healthy without the API key its seal hook delivers.

Two tests hold the two sides. tests/_ce/modules/app/post_hook_after_ready_test.go fails if ce runs post_install before the replicas are up or for an install that ended waiting. tests/unit/catalog/nextcloud_notify_push_idle_test.go fails if notify-push stops reading healthy without its binary, or gives up waiting for it.

An update of a stopped deployment leaves its post hooks for the next start

Section titled “An update of a stopped deployment leaves its post hooks for the next start”

shc stop then shc update is a normal window step: lock the app, change its config, start it. The update renders and applies the new config and leaves the deployment stopped. It does not run the part of its post_update phase that needs a service the operator stopped, because nothing could exec into it:

  • the post_update chain, when one of its actions needs a stopped service: a context: service job on it, a start or stop of it, or an http job or a templated service when every service is stopped (a context: container job needs none);
  • the consumer plugs’ jobs.post_deployment after the chain, when the chain waits or every service is stopped.

The update says so on its result line with X200121HKDEFR, naming the jobs and shc start <stack>, and records them. The next operator shc start (or shc start -S of the services they need) runs them once the deployment reads ready, keeps the values they mint as an update does, and reports them under its result with X200122HKDRAN. They run once. A start that leaves a service they need stopped runs none and says X200124HKDWAI. A job that fails, or one that changes a value its services read when they were rendered, fails the start’s answer with X500908HKFAIL naming shc update; the services stay up. An update whose chain runs forgets anything an earlier update left.

Two things cannot wait for a start, so the update refuses them before it changes anything: a plug vapp: instance it would make in a deployment whose every service is stopped (X409912HKNDEF), and an update that cannot read which services the operator stopped (X503916HKSTOP). tests/_ce/modules/app/deferred_hooks_6993_test.go holds fizzy’s and librechat’s shapes.


Hooks are declared in app.yaml as a map from hook name to an ordered list of steps. A step is a job name (not an inline command) or a stop: / start: list of service names. Job names refer to job definitions loaded from *.job.yaml files in the same app directory.

stop: stops those services so a one-off job can own their data. start: starts them again. Neither changes a replica count; replicas come from --scale on install and update. Where the process runs replica projects, stop and start go through that process’s pod runner. Otherwise they are compose stop / compose start of the named services on the deployment’s project. A hook run that fails starts again every service it stopped and did not start. A run that finishes, or a job that emits ::shc-hook:halt::, does not start those services again. The scale: hook action is gone; a step that names it is refused.

apps/myapp/app.yaml
hooks:
post_install:
- init-db
- notify-admin
pre_update:
- stage
- stop: [db]
- migrate
- start: [db]
pre_backup:
- flush-cache
pre_restore:
- drop-existing-schema
apps/myapp/init-db.job.yaml
name: init-db
context: service
service: postgres
command: |
psql -U {{ vars.admin_user }} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
timeout: 60
on_failure: abort

Jobs run sequentially in the order they’re listed. One job fails → the hook is considered failed, but see “failure behaviour” below.

A job that drops or replaces whatever schema its datastore holds, a first-boot seed ceremony, says so with seeds: true. In post_install and post_update, SHC refuses that job while the deployment, or a provider it binds, carries a standing restore seal, so the install that follows a restore never wipes it. shc backup clear-restore-seal lifts the seal once the restored data has been checked. A job whose command drops a table or runs a framework’s destructive migration is refused the same way without the declaration.

apps/myapp/app-init.job.yaml
name: app-init
context: service
service: app
seeds: true
command: php bin/console app:install --force

A job that restarts a service it reconfigured

Section titled “A job that restarts a service it reconfigured”

A service that reads a file only at start keeps running the old content when a job rewrites that file. The job asks for the restart itself by printing ::shc-hook:restart:<service>:: on a line of its own, and restarts: on the job names every service it may restart. SHC stops and starts each named service once, after the job ended 0, the same way a stop: and start: step would. A job that changed nothing prints no marker and restarts nothing. A failed job restarts nothing. A marker for a service restarts: doesn’t declare fails the job with X500127HKRSTU before anything restarts. The rule holds for *.job.yaml hook jobs and for a plug’s jobs.post_deployment jobs alike, and the catalogue test TestEveryHookRestartIsDeclared holds every printed marker to its job’s restarts:. SHC reads restarts: from 0.1.28, and 0.1.27 and older refuse the field at install and update, so a package that uses the marker declares requires.shc: "0.1.28" or later in app.yaml. The catalogue check fails a package that uses it without that.

Socket jobs.integration jobs may also declare restarts: and print the marker since 0.1.29. They restart services of the provider deployment, after the job succeeds and before the next integration job runs. A socket job that declares restarts: requires requires.shc: "0.1.29" or later, even if it prints no marker on a replay: 0.1.28 and older refuse the field with X400361INTFLD. TestEveryHookRestartIsDeclared checks socket markers against the job’s declaration and the provider app’s compose services too.

apps/myapp/myapp.plug.yaml
jobs:
post_deployment:
- name: rewrite-config
context: service
service: app
restarts: [app]
command: |
render-config > /etc/app/config.json.next
if cmp -s /etc/app/config.json.next /etc/app/config.json; then
rm -f /etc/app/config.json.next
else
mv -f /etc/app/config.json.next /etc/app/config.json
echo "::shc-hook:restart:app::"
fi

netbird’s netbird-rewrite-management-json is the shipped case. It asks for the restart when management.json is newer than the management process, because compose’s init container can write the new file before the job runs.


A job’s context field selects where it runs. The available contexts (from JobContext in job/types.go):

ContextRuns inTypical use
serviceAn exec on an already-running compose serviceSeed a database after post_install
containerA new container that runs once, of its image, or with no image of its service’s own definitionRun migrations in a throwaway container (pre_install before the deployment is up)
httpA host-side HTTP request to urlPing a webhook, hit an admin API
graphqlA host-side GraphQL mutation against urlCall a GraphQL admin endpoint
pipelineA sequence of steps (each a sub-job)Multi-step provisioning
playwrightA Playwright browser script against urlAutomate a browser-based setup UI

A service job runs on every runtime. A container job depends on the runtime of the node that runs it:

Runtimecontainer job
docker, podmancompose run --rm in the replica’s project. On podman the job starts through SHC’s exec helper, as a service does, so it holds docker’s supplementary groups and default capabilities and becomes its user: the way docker does. Its image is pulled first when the node lacks it.
incusA project of its own, shcjob-<run>-<job>, holding the one job. The incus executor brings it up, reads it back until it has ended and removes it with its project, whatever the outcome. The job’s output is the instance’s console log, so stdout and stderr arrive as one stream. container_name is not carried over: incus names its own instances.
appleA project of its own, as on incus, whose one container starts held. Its output is the container’s log.

Why apple holds the job: a job on the fabric runs at a pin the node’s DNS and policy already know. apple assigns a container’s address itself as it starts the container and refuses a pinned one (ipv4_address is a refused key on apple, see compose keys). So the job’s first process waits before its command until the address apple gave it is the job’s pin, and only then runs. A container that holds no address after 30 seconds fails the job with X500359FABPIN before its command runs.

A job’s volumes go through the mount policy a service’s do. A named volume becomes the deployment’s data bind, and the fields it names (handover:/srv/handover:0750:1000:1000, noinit) are applied on the node that runs the job before its container starts, as they are for a service’s mount (compose keys): while the directory is new or empty, on the directory itself. A job copies nothing of its image into a volume. A mount that names no fields is made if it is missing and left as it is.

A container job that names a service and no image runs as one new container of that service’s own definition, as compose run --rm --no-deps makes one, with the job’s command. Nothing of the service is written twice:

medusa-migration.job.yaml
name: medusa-migration
context: container
service: medusa
after: [valkey]
before: [medusa, medusa-worker]
command: ["npx", "medusa", "db:migrate"]
  • It takes the service’s image, environment and env_file, volumes and binds, configs, secrets, user, working_dir, capabilities and resource limits, as the replica’s document holds them.
  • It takes none of its ports, healthcheck, restart, depends_on, container_name, labels or network aliases: a job never answers for the service. It joins the replica’s networks as every container job does, at a fabric address of its own.
  • The job gives the command, and may give an entrypoint, a user, a working_dir and environment, whose keys win over the service’s. An exec-form command runs under the service’s own entrypoint, as compose run does. It names no volumes: a job that needs another mount names its image instead.
  • It runs on the node that holds the service’s lowest replica, whose mounts and networks are made ready while that replica stays stopped, so the service need not run: a migration names it in before:. A scaled service runs the job once, from replica 0. For a group member name the member (postgres-1), never the group.

Not every context is valid for every hook: pre_install can’t exec into services that don’t exist yet. The mapping is defined in allowedContexts in hooks.go:

HookAllowed contexts
pre_installhttp, container
post_installhttp, container, service
pre_updatehttp, container, service
post_updatehttp, container, service
pre_uninstallhttp, container, service
post_uninstallhttp, container
pre_backuphttp, container, service
post_backuphttp, container, service
pre_restorehttp, container, service
post_restorehttp, container, service

Jobs using a disallowed context are skipped with a warning (X500908HKFAIL): the hook is not aborted, the invalid job simply doesn’t run. The helper _validate_job_context enforces this.

Why service is disallowed in pre_install and post_uninstall: at pre_install the deployment hasn’t started yet, so there’s no container to exec into; at post_uninstall the containers are gone.

pre_restore runs while the deployment is still up, before the restore stops it, so a service job can ask the running app whether the restore may go ahead. A job that prints ::shc-hook:refuse:: <why> and exits non-zero stops the restore before anything is stopped, backed up or written (X409876RSTHKR, carrying its line); one that fails without the marker is a warning and the restore goes on. The hooks fire from the deployment’s stored package on a node that holds its replica and no app tree. A restore into a target that does not run, a recover onto an absent target or the target of a clone or move, fires no pre_restore: nothing runs for it to ask. Its post_restore runs once the replicas start.


The HookExecutor assembles a variable dict before handing the job to JobExecutor:

  1. vars.yaml: the merged variable dict persisted during install (see the install flow). Exposed to templates as vars.*.
  2. Secret references resolved: any ref+vault://... strings in vars.yaml are resolved before the job runs, so commands see real values, not URIs. This is specifically so bootstrap hooks (e.g. hashing an admin password) see the real password. See loadDeploymentVars in hooks.go. Secret resolution uses strict=False: an unresolvable ref leaves the URI literal in place rather than aborting.
  3. Job-level constants: stack_name, tenant_name, environment, node_id, deployment_dir passed through the JobExecutor constructor.

In job YAML, reference them with Jinja:

command: |
echo "installing {{ stack_name }} in {{ environment }}"
psql -U {{ vars.admin_user }} -c "SELECT 1;"

Each job has an on_failure field (see OnFailure enum). But note the hook-level contract is independent:

  • on_failure: abort: if the job fails, abort the hook (subsequent jobs in the same hook list don’t run).
  • on_failure: continue: log the failure, run the next job anyway.

A hook failure (a job exiting non-zero, or any other error inside the hook chain) is logged as a warning (X500908HKFAIL) and does not propagate to the caller.

That means: the pipeline treats hooks as best-effort, with exactly one exception. A failed post_install hook does not fail the install. A failed pre_backup hook does not fail the backup.

The one exception: pre_update gates the update

Section titled “The one exception: pre_update gates the update”

pre_update is the phase that runs while the old containers are still serving, which makes it the only place an app can answer “may this version replace the one on disk?”. A refusal there is therefore fatal to the update: the pipeline abandons it with X409005HKGATE before anything is pulled, recreated or scaled, and the deployment keeps serving its previous version.

What is and is not restored:

  • Containers: untouched. Nothing was brought down or up.
  • Rendered artifacts: the deployment dir holds the NEW compose.yaml, app/ and vars.yaml, the same as after any other failure between the render and compose-up. Re-run the update once the hook’s complaint is answered; nothing needs repairing first.
  • The manifest sidecar: still records the version the deployment is running, because it is written only after a successful compose-up.

A context: service pre_update job is SKIPPED, not failed, when its service has no running container (X200912HKSKIP). docker compose exec into a stopped service cannot succeed, and the update path is also how a deliberately stopped deployment is re-rendered: shc stop then shc update, and both legs of a move’s fast path, whose stop precedes the re-render and its rollback. A gate that must hold with the deployment down belongs in a context: container job (those still gate) or in compose itself.

Belt and braces is still the recommendation

Section titled “Belt and braces is still the recommendation”

Two of the three packagings that gate an update (postgres, the major-version ladder, and jellyfin, the one-way library migration) keep a second copy of their refusal as a compose service whose exit code the app’s depends_on: service_completed_successfully reads. Keep doing that for anything irreversible: the compose copy covers the stopped-deployment window the service-context hook skips, and it fires on every start, not only on the ones SHC drove as an update.

Other phases have no gate. If pre_install is expected to provision something the container depends on, write the container to retry or refuse to start until the precondition is met. Don’t rely on the hook failing the install.


Jobs that report a value: read_only and sensitive_output

Section titled “Jobs that report a value: read_only and sensitive_output”

A job no hook names runs when something asks for it: shc job run <job>, a schedule, or the shc_job_output data source of the Terraform provider. Two keys say what such a job does.

read_only: true says the job changes nothing. shc_job_output can start a job at plan time, so it refuses one that doesn’t declare this. SHC doesn’t inspect the command: the declaration is your promise.

sensitive_output: true says the job prints a secret. Its stdout, stderr and error are sealed under the vault’s key on its run’s row, and reading them takes stack:manage on the deployment. stack:read still sees that the run happened and how it ended, and stack:operate can start it without learning its answer.

name: list-projects
context: service
service: web
read_only: true
sensitive_output: true
command: ./list-projects

A job that declares sensitive_output has to run on demand in service context. A container job’s stdout is its container’s log, which the collector ships and stack:read reads, so it’s refused with X400028JOBSNS. A hook or start job’s output reaches its caller and the daemon’s log through the failure text, so it’s refused with X400029HKSENS. With the vault locked, the run is refused before it starts with X503919JOBSEL.


JobLoader scans the app directory (paths.app_dir(tenant, stack, environment)) for files matching *.job.yaml. Each file is parsed as a single JobDefinition. The lookup by job name is linear but cached per hook invocation.

Job files are copied into the deployment during the install flow alongside the rest of the app sources.


Run-once work is a job, never a service. A job file that sets after:, before: or both is a start job. It orders the deployment’s start, and no hook names it.

FieldValueMeaning
afterlist of service or start job namesThe job runs once a replica of each service reads ready (running, and healthy where it has a healthcheck) and each job named ended 0.
beforelist of service or start job names, a service optionally as name:stoppedThose services start only once this job ended 0. A job named here runs after this one. On an update, a service named name:stopped has its old replicas stopped before the job runs.
seed.job.yaml
name: seed
context: service
service: counter
after: [counter]
command: redis-cli setnx visits 0
migrate.job.yaml
name: migrate
context: container
image: myapp:{{ vars.version }}
after: [db]
before: [web]
command: ["python", "migrate.py"]
checkpoint.job.yaml
name: checkpoint
context: container
service: media
before: [media:stopped]
command: ["media-server", "--checkpoint-library"]

What SHC does with them:

  • An install and each update run every start job once for the deployment, never once per replica. A service job execs into one replica of its service, on whatever node holds it.
  • A replica of a service a job’s before names waits, on any node, until that job’s run ended 0. The order is the same one SHC keeps between pods for depends_on.
  • A job that does not end 0 fails the install, which rolls back. An update runs its start jobs before anything of it starts, against the replicas already running, and one that fails ends the update before its spec commits.
  • A job runs as long as its own timeout lets it, counted from its command’s start (600 s when unset; for an http job, its attempts and the pauses between them). A container job’s bring-up (its image, its container’s create and start) has its own bound: 15 minutes, or the timeout when longer. Ten more minutes cover reaching its node and answering. The 15-minute settle window bounds how long an install or update waits for replicas to read ready; it does not count the time a start job runs, nor a replica that waits for a running job’s before. A job still running past its own budget fails the install, naming the job.
  • A job whose after names a service the same update adds, or that runs after such a job, runs as an install runs it. The update commits its spec, the new service starts, and the update waits until the service reads ready and then runs the job. If the job fails, the update fails after its commit, and the services its before names do not start.
  • A job whose before names a service as name:stopped is for work that must not run beside the old server, such as a checkpoint or an archive of its data. On an update SHC stops that service’s old replicas, runs the job once they read stopped, and starts the service on the new version only after the job ended 0. If the job fails, the old replicas start again as they were and the update fails before its spec commits. On an install there is no old replica, so the job runs as any start job. A service its operator stopped stays stopped.
  • A move and a failover keep the run the deployment has, so nothing runs again. A restore keeps it while the cluster still holds the deployment’s spec, and runs the jobs on a cluster that does not. A restart never runs one.

The install refuses a start job with a context other than service, container or http, a service job whose after does not name its own service, a container job with neither image nor service, one that runs as its service and names volumes, a job named apart from its file, a before entry that is neither a name nor name:stopped, a start job written name:stopped, a name the app does not have (a service a job runs as included), a job named like a service, a start job a hook also names, and an order that never begins (a job before what it runs after, directly or through depends_on). The error names the job that failed. shc validate checks the names.


app.yaml
hooks:
post_install:
- create-schema
create-schema.job.yaml
name: create-schema
context: service
service: postgres
command: |
psql -U {{ vars.admin_user }} -d {{ vars.db_name }} -c "
CREATE TABLE IF NOT EXISTS users (id SERIAL PRIMARY KEY);
"
timeout: 60
on_failure: abort

Run a migration in a throwaway container before the deployment starts

Section titled “Run a migration in a throwaway container before the deployment starts”
app.yaml
hooks:
pre_install:
- migrate
migrate.job.yaml
name: migrate
context: container
image: myapp:{{ vars.version }}
entrypoint: ["python", "migrate.py"]
networks:
- shc_{{ tenant_name }}
volumes:
- "{{ deployment_dir }}/data:/data"
timeout: 300
on_failure: abort
notify.job.yaml
name: notify
context: http
url: https://hooks.example.com/deploy
method: POST
headers:
Authorization: "Bearer {{ vars.webhook_token }}"
body:
stack: "{{ stack_name }}"
tenant: "{{ tenant_name }}"
event: "installed"
max_retries: 3
retry_interval: 10

Flush a cache before backup so the snapshot is consistent

Section titled “Flush a cache before backup so the snapshot is consistent”
app.yaml
hooks:
pre_backup:
- flush-cache
flush-cache.job.yaml
name: flush-cache
context: container
image: redis:7
entrypoint: ["redis-cli", "-h", "redis", "BGSAVE"]
networks:
- shc_{{ tenant_name }}
timeout: 30
on_failure: continue

on_failure: continue because a cache that can’t flush is not worse than not flushing it; the backup should still proceed.


  1. Check the daemon log for X500908HKFAIL: that’s the hook failure event class.
  2. shc logs -s <stack>: the container-context job’s output.
  3. The JobRunModel table has one row per executed job: shc job history -s <stack> shows recent runs.
  4. For service-context jobs, docker logs on the service container will show the exec’d command’s output.

Most common cause: a job is using a context the hook doesn’t allow (e.g. pre_install with context: service). Check for X500908HKFAIL with context 'X' is not allowed for hook 'Y' in the message.

If {{ vars.admin_password }} appears literally in a job’s command, the variable isn’t in vars.yaml (typo or missing from the merged vars). If a ref+vault://... URI appears literally, the vault provider couldn’t resolve it (see vault provider logs).