Credential reconciliation

Make the vault the single source of truth for a credential, and have the app converge to it on every deploy — so the secret can be rotated, vault rm’d, or restored from a mismatched backup without locking the app out.

SHC mints app credentials as autogen vault secrets (ref+vault://admin_password?autogen) and renders them into the container’s env. For a stateless value that the app re-reads every boot, that’s the whole story — change the vault value, redeploy, done.

But many apps persist the credential into durable storage at first init and then ignore the seed env var forever after:

• A database bakes its root password into the data directory on initdb (POSTGRES_PASSWORD, MARIADB_ROOT_PASSWORD, MONGO_INITDB_ROOT_PASSWORD are honored only when the volume is empty).
• An app seeds an admin account row into its database the first time it boots (keycloak, nextcloud, sonarqube, gitlab …) and authenticates against the stored hash thereafter.

The credential and the data it protects become a matched pair. If the vault value later changes but the stored copy doesn’t, every reader using the new value fails against the old hash — a lockout. (This is exactly the keycloak admin-password drift class: container seeded one value, vault held another, nothing could authenticate.)

Two ways to keep the pair consistent:

1. Pin the secret — never let it change while the data lives. This is the default SHC behavior: uninstall never deletes vault rows, so a reinstall reattaches with the same password. Simple, but the secret can never be rotated and a restore must carry its matching secret.
2. Reconcile — on every deploy, force the stored credential to equal the current vault value. The vault becomes authoritative; the app self-heals. This is the better invariant when the app supports it.

All three must hold:

1. The credential is owned by this app — its own autogen var — not a DB password handed to it by an integration/plug. (A consumer’s DB password is the provider’s concern: the postgres/mariadb vsocket reconciles it with ALTER USER when the edge is (re)established. Don’t reconcile it from the consumer.)
2. It is written into durable storage at first init (the app’s own data dir/volume, or an account row in a database).
3. On later boots the seed env var is ignored (the stored value wins), so a vault change does not propagate on its own.

If instead the value is re-read from env every boot (e.g. a Rails SECRET_KEY_BASE, redis requirepass), nothing to do — a redeploy already picks up the new value.

Reconciliation only works if you can set the credential without already knowing the current one. That splits credential-bearing apps in two:

The app exposes a reset that runs with local/admin authority and does not require the old password:

• Databases — connect over the in-container unix socket (peer/trust auth, no password) and run the idempotent reset:
  • postgres: ALTER USER <user> WITH PASSWORD '<vault>';
  • mariadb/mysql: ALTER USER '<user>'@'%' IDENTIFIED BY '<vault>';
  • mongodb: db.changeUserPassword('<user>', '<vault>') (localhost exception)
• Apps with a server-side admin CLI that bypasses login (runs with DB access inside the container):
  • nextcloud: occ user:resetpassword --password-from-env admin
  • paperless / Django: python manage.py changepassword admin (or a shell user.set_password(...))
  • gitlab: gitlab-rails runner "User.find_by(username:'root').update!(password: ENV['ROOT_PASSWORD'])"
  • frappe: bench --site <site> set-admin-password <vault>

These converge totally — even from a fully-diverged password. Bake the reset in (below).

The reset runs through the app’s own admin API/CLI, which you must log into first. The reset itself is live and real — keycloak’s kcadm set-password, a Grafana/PMM PUT /api/user/password — but it needs a working credential. So you can reconcile a password you can still authenticate with (drift you can recover), but not one fully reset out from under you: you’d need the value you no longer have.

Examples: keycloak (kcadm set-password resets live but needs a login; with every admin credential lost, kc.sh bootstrap-admin needs all nodes stopped and Argon2 blocks a DB rewrite), sonarqube, metabase, grafana/pmm, immich, windmill, peertube, fleet, penpot.

For Class B, keep the anchor credential pinned (don’t rotate the account the reconcile job logs in as), and treat a hard divergence as an operator recovery — restore a backup whose secret matches the data, or the app’s stopped-server recovery. apps/keycloak/keycloak-init.job.yaml is the reference: it logs in with the vault value, falls back to the bootstrap value and rotates, and on a full loss fails loudly with the real recovery steps.

Promote B → A with a pinned anchor. Give the app a second admin whose secret is never rotated; the reconcile job always authenticates as that anchor, then resets the rotatable admin via the API/CLI. The anchor login never drifts, so reconciliation always succeeds — even after the primary credential is fully reset — making the rotatable admin freely rotatable. (Cost: a second managed account and the discipline of never rotating it.)

Two vectors — use both as appropriate:

Before starting the app, force the credential from the env. Guarantees convergence on every start, even a bare docker run outside SHC:

# entrypoint, before exec'ing the server — idempotent, needs no old password
psql -U postgres -h /var/run/postgresql \
  -c "ALTER USER ${POSTGRES_USER} WITH PASSWORD '${POSTGRES_PASSWORD}';"

The reset path must be current-credential-independent (socket/peer auth, or a CLI with DB access). This is the strongest form — the image enforces it.

When you can’t change the image, run the reset as a post_deployment lifecycle hook (see lifecycle-hooks.md). Name it <app>-reconcile.job.yaml, or fold it into the existing <app>-init.job.yaml:

jobs:
  post_deployment:
    - name: reconcile-admin-password
      context: service
      service: <app>
      command: |
        set -eu
        occ user:resetpassword --password-from-env admin
      env:
        OC_PASS: "{{ vars.adminPassword }}"

Rule of thumb: bake it into containers you build; use a job for third-party images. Both make the vault authoritative.

First-pass classification of the 44 apps that render a self-owned credential (plus postgres, audited separately). Treat the A/B line as the actionable split; a few edge apps (SQL-with-hashing resets) sit between.

Class A — bake in / add a reconcile job (clean, total): postgres, mysql, mariadb, mongodb (database root, via socket); nextcloud, paperless, gitlab, frappe, ciso-assistant, keygen, monica, penpot, plane (rabbitmq), greenbone, fusionpbx (server-side CLI / privileged SQL).

Class B — keep the secret pinned; reconcile only recoverable drift: keycloak, sonarqube, metabase, pmm (grafana), immich, windmill, peertube, fleet.

No reconciliation needed:

• env-every-boot (re-read each start): blockscout, docuseal, gitea, highlight, hyperswitch, minio, postiz, shlink, wordpress.
• integration-provided (the provider’s vsocket reconciles the DB cred): affine, defectdojo, dependency-track, hockeypuck, kutt.
• none (no persisted self-credential): ivatar, jellyfin, matrix, netbird.

The audit is a starting map, not gospel — confirm an app’s reset path against its image before wiring a job. A credential whose only reset is an authenticated API call is Class B no matter where it’s stored.

With Class A reconciliation in place, the vault is authoritative into the running app, not just at render time:

• Secrets become freely rotatable (security).
• Uninstall could safely purge the secret (a reinstall reconciles) — the “secret must outlive the data” coupling relaxes for these apps.
• A restore against a new secret self-heals.

Class B apps keep the coupling: pin the secret, lean on backups for recovery.