Vault & secrets management
SHC’s vault is the single place where every secret used by a
stack lives: database passwords, API keys, OIDC client secrets,
webhook tokens. Secrets are referenced from app.yaml, vars.yaml,
and compose.yaml using a ref+vault:// URI; SHC resolves and
injects the real value at render time.
SHC supports 11 backend providers for the vault, so you can keep
the default provider (encrypted in SHC’s state database), or delegate to an external
secret manager (AWS Secrets Manager, Azure Key Vault, HashiCorp
Vault, GCP, Bitwarden, Doppler, Infisical, 1Password, pass, SOPS).
Source of truth:
modules/vault/service.go,
modules/vault/crypto.go,
modules/vault/providers/,
core/config/secrets.go (ref URI parser).
For the full authorization / scoping model see architecture/security.md.
Quick mental model
Section titled “Quick mental model”flowchart LR compose["compose.yaml<br>PASSWORD:<br>ref+vault://.."] --> render["render pipeline<br>resolve_ref_async"] render --> vault["vault<br>provider.get"] vault --> injected["real plaintext<br>injected into compose env"]- Secrets are never stored in plaintext in
app.yaml,vars.yaml, deployment files on disk, or compose before render. - Only the reference URI is persisted; the actual value lives in the vault provider.
- References carry a scope (per-stack, per-environment, per-tenant, or global) so the same URI in different stacks resolves to different values.
The default provider: local
Section titled “The default provider: local”Every SHC install gets a local vault provider out of the box: values are
envelope-encrypted at rest in SHC’s state database. One Argon2id
key-encryption key is derived from the master password at each unlock and held
in memory; every secret carries its own random AES-256-GCM data key, wrapped
by it. Stored values are tagged v2:.
There is exactly one at-rest format. The PBKDF2/Fernet format used before
v0.0.16 has no reader; see
the vault module for the
retirement and the one-way make migrate step it requires.
Tenant isolation is database-level. Each tenant’s secrets are stored under its tenant row; reads are guarded by the same scope-matching rules the permission decorator uses; see security.md.
First-time setup
Section titled “First-time setup”There is no separate vault-init step. The master password is set when the node is initialized, and only a verifier hash (PBKDF2) is stored in the state database; the password itself is never persisted.
shc init --vault-password-file /root/vault.pw --yes # non-interactiveshc init # prompts for it--vault-password-file expects mode 0600 and takes precedence over
--vault-password and SHC_VAULT_PASSWORD; pass - to read stdin.
Unlocking on daemon start
Section titled “Unlocking on daemon start”SHC refuses writes to the vault while locked. On every daemon start
shcd opens the sealed auto-unlock credential once and unlocks with
it when the file is present. A missing file is the follower posture: the
vault stays locked until an operator runs shc vault unlock (or, on a
multi-node cluster, until a peer pushes the unlock).
shcd never reads SHC_VAULT_PASSWORD. That variable is the retired
delivery path. If it is set in shcd’s environment, startup refuses with
X400975VLTRFE and does not fall back to the variable or to the file.
SHC_VAULT_PASSWORD remains a CLI input for shc init and shc vault
only; those are a different process from shcd.
Source:
subsystems/impl/vault.go.
The sealed auto-unlock credential
Section titled “The sealed auto-unlock credential”The cold-start node keeps the master in one file, sealed with
systemd-creds under the credential name vault-unlock:
# /etc/systemd/system/shc.service.d/vault-autounlock.cred (chmod 0600, root:root)<systemd-creds output; no plaintext>It is sealed to the node’s credential host key
(/var/lib/systemd/credential.secret), and to its TPM2 as well when the
node has a usable one. It is bound to no TPM PCR: a PCR 7 binding breaks
on a Secure Boot or firmware update and would strand the node that
breaks the cluster’s cold-start deadlock. No unit line names the file.
shcd opens it itself at boot with systemd-creds decrypt, so the
plaintext exists only in shcd’s memory. shc.service runs
systemd-creds setup before shcd starts, which creates the host key
once.
A sealed credential that will not open (the host key or the TPM it was
sealed to changed, or the file is damaged) does not stop shcd. The daemon
comes up with the vault locked and logs a warning naming the file. Unlock it with
shc vault unlock, which seals a fresh credential, and the next restart
auto-unlocks again. The vault.autounlock_at_rest row of shc doctor run -c vault reports
a credential that no longer opens before a restart finds out.
Three genesis paths plant it, and they plant the same file: the
terraform provider (writeVaultAutoUnlockDropIn), shc init when it is
given the master password (--vault-password-file, --vault-password,
or stdin), and the daemon when cold-start moves onto a node
(plantLocalColdStart). The master reaches systemd-creds on stdin,
never in argv, and every seal is opened again and compared before the
file is written. On launchd the same paths write a System keychain item
instead (see below). A resumed shc init (the node
is already bootstrapped) re-plants it only when it unlocks the vault
itself and the vault accepts the password. When the vault was already
unlocked, or the unlock is refused, it plants nothing and, if the
credential is missing, prints the repair:
shc vault coldstart replant --node <node>. shc vault rotate and
shc vault unlock keep an existing credential in step and do not create
one.
A Mac node: the System keychain
Section titled “A Mac node: the System keychain”On launchd the credential is a generic-password item in the System
keychain (/Library/Keychains/System.keychain, service
io.selfhostedcloud.shc.vault-unlock, account shcd). No other account
can open it without an administrator’s authorization. The root launchd
shcd reads it at boot. shc init and the cold-start replant write it
through /usr/bin/security on stdin, so the password never appears in a
process list, and read it back before it counts. Rotation and every unlock
keep an existing item in step and never create one on a follower.
A Mac that has a /etc/shc/vault-unlock file (the plaintext form before
the keychain) still unlocks from it once. The next unlock then moves the
password into the keychain item and drops the file once the item reads
back. vault.autounlock_at_rest flags the file until then.
A node whose systemd cannot seal
Section titled “A node whose systemd cannot seal”systemd-creds arrived in systemd 250. Debian 12 (252), Debian 13 (257),
Ubuntu 24.04 (255) and RHEL 9 (252) all seal. On an older systemd
(Ubuntu 22.04 ships 249) SHC writes no credential at all, never a
plaintext one: shc init and the provider warn (X200052VLTSKP), and
that node’s vault needs shc vault unlock after every restart.
vault.autounlock_at_rest keeps saying so.
Upgrading a v0.1.21 cold-start node
Section titled “Upgrading a v0.1.21 cold-start node”v0.1.21 kept the master in plaintext beside a unit line that named it:
[Service]LoadCredential=vault-unlock:/etc/systemd/system/shc.service.d/vault-autounlock.envand vault-autounlock.env held the raw password and a newline. The
upgrade converts it in place with no step from you. The first start of
the new shcd still unlocks from that file. The repair that follows every
unlock then seals the master and opens the seal back. It removes the
.conf, reloads systemd, and removes the .env last. At every step,
both the unit on disk and systemd’s cached copy name a file that is
there. A v0.1.21 .conf naming a missing .env failed the unit with
243/CREDENTIALS. The sealed form names nothing, so that failure is
gone. On a node that cannot seal, the v0.1.21 pair stays, and rotation
keeps it in step.
Do not put the password in the unit environment.
Environment=SHC_VAULT_PASSWORD=… and EnvironmentFile= both deliver
it through the process environment. systemd publishes a unit’s resolved
environment as a queryable D-Bus property, and /proc/<pid>/environ
keeps that copy for the life of the process. A unit that still sets
SHC_VAULT_PASSWORD makes shcd refuse to start (X400975VLTRFE). The
package ships the guard at
/usr/lib/shc/scripts/vault-password-not-queryable.sh; run it on any
node to check.
On a multi-node cluster only the genesis node holds the credential: followers come up locked and pull the unlock from an already-unlocked peer.
The password is the credential’s bytes, minus one newline
Section titled “The password is the credential’s bytes, minus one newline”The sealed plaintext is the password and one trailing newline. shcd trims exactly that newline and uses the rest as the password.
shc vault rotate still refuses a master that contains whitespace, a
line break, a quote, a backslash, or a percent sign (X400965VLTDRI).
Choose a master inside that set (random_password with
special = false is what both live estates use).
Manual unlock / status / lock
Section titled “Manual unlock / status / lock”shc vault # prompts for password, unlocks the vaultshc vault unlock # explicit alias of bare `shc vault`shc vault --password "$PW" # unlock non-interactivelyshc vault status # shows initialized + locked stateSource:
modules/vault/commands/main.go.
Rotating the master password
Section titled “Rotating the master password”The master password derives the KEK that wraps every per-value data key. Rotating it re-wraps every encrypted value (the payload ciphertext itself is untouched) and flips the replicated password verifier in the same atomic batch:
shc vault rotate # prompts for old + newDo this after restoring from a backup from a different cluster, or when the old password may have leaked. Other provider types (AWS / HashiCorp / …) handle this in their own dashboards; SHC does not try to proxy it.
A rotation makes the unlock-key escrow receipt stale. Escrow the new master and record it again (next section).
Escrowing the unlock key and the backup key
Section titled “Escrowing the unlock key and the backup key”Two keys decide whether an estate can be recovered after it is lost:
- the unlock key, the vault master password. On the cold-start node it is in the sealed auto-unlock credential, which opens only on that node and is in no backup archive.
- the backup key, the restic repository key
backup.passwordnames.shc initseals it into this vault and nowhere else. Every off-site archive needs it to open, and it does not change when the vault rotates.
Lose the estate with both inside it and every archive stays sealed. So copy each one out of band, somewhere outside the estate: a password manager, a printed sheet in a safe, a CI variable on another system. Keep the two apart if you can, so no single copy opens everything. SHC never holds, moves or checks that copy. You record that it exists:
shc vault escrow record --what unlock-key --where '1Password, vault Infra, item SHC master'shc vault escrow record --what backup-key --where 'printed sheet, office safe' \ --note 'repository URL and bucket account on the same sheet'shc vault escrow status # each key: ok, stale, missing or not neededshc vault escrow verify # same table, exits non-zero unless every key is okA receipt holds the label, the note, the node and principal that
recorded it, and the key’s generation. It never holds the key: a
--where or --note that contains the master or the backup key is
refused (X400985VLTESK), because receipts are replicated into every
database backup.
The generation is what makes a receipt go stale:
| Change | Stale receipt |
|---|---|
shc vault rotate | unlock-key |
shc backup key rotate, or any write of the backup key | backup-key |
Re-escrow the new key, then run record again; a second record for the
same key replaces the first. The backup key needs no receipt when
backup.password is unset or points at another secret provider, which
holds it off the estate.
shc doctor runs vault.unlock_key_escrow on every node. It fails while
a key that needs a receipt has none or a stale one, and its hint names
the record command to run.
Reference syntax: ref+vault://
Section titled “Reference syntax: ref+vault://”Wherever you’d write a plaintext secret, write a reference instead:
adminPassword: "!ref+vault://admin_password"
# With explicit scopeapiKey: "ref+vault://shared_key?scope=tenant"
# Auto-generate on first installsessionSecret: "ref+vault://session?autogen&length=32"
# Custom charsetdbPassword: "ref+vault://db?autogen&charset=alphanum&length=24"URI grammar
Section titled “URI grammar”ref+<provider>://<key>[?<params>]
params: scope=<base|deployment|environment|local|stack|tenant> autogen generate a value if missing length=<n> autogen length (default 32) charset=<alphanum|ascii> autogen charset (default printable) <custom>=<value> forwarded to the providerParser:
core/config/secrets.go (ParseSecretRef, SecretRef).
Scopes
Section titled “Scopes”| Scope | Key namespace | Typical use |
|---|---|---|
deployment (default) | (tenant, stack, environment) | Per-install values: this is almost always what you want |
environment | (tenant, environment) | Values shared across all stacks in one environment |
stack | (tenant, stack) | Shared across environments of one stack |
tenant | (tenant,) | Tenant-wide values (an API key for a single SaaS account) |
local | Per-node | Node-local secrets (rare; see node-agent config) |
base | Global | Cluster-wide defaults; use sparingly |
Inline refs in long strings
Section titled “Inline refs in long strings”A ref embedded in a larger value is resolved in place:
connection_string: "postgres://user:ref+vault://db_pass@host:5432/db"The pattern matches ref+vault:// followed by non-whitespace characters (from
secrets.go), so the ref ends at whitespace,
quotes, or closing braces/brackets.
Auto-generation
Section titled “Auto-generation”?autogen tells SHC: if this key has no value yet, generate one
and store it. Every update, restart and redeploy of the deployment
reuses the stored value, so auto-generated admin passwords stay stable
across upgrades and restarts. An uninstall purges the deployment’s keys
with its config. A later install under the same name is a new
deployment and mints new ones, and shc recover -b with the
uninstall’s safety backup brings the old ones back with the data.
Never commit ?autogen keys for user-facing credentials you need to
hand back to the user; use shc vault get <scope> <key> to retrieve
the generated value after install.
OIDC client secrets
Section titled “OIDC client secrets”On install or update, an autogen vault ref in vars that supplies
oidc.plug.yaml’s data.clientSecret gets a credential=deployment
parameter. SHC persists the marked ref, not its resolved value. It reads
deployment, stack, then environment scope. A row at any of those tiers wins;
otherwise SHC creates a deployment-scoped row. Tenant and global rows are
excluded, including when the ref previously requested a broader mint scope.
Subsequent renders and read-only checks use the same restricted read.
Packages can also declare ?autogen&credential=deployment for another
per-deployment credential. Ordinary refs and non-autogen OIDC refs retain
their existing cascade. Existing deployment rows are reused.
If a deployment’s OIDC client secret came from a tenant or global row, whether in one environment or shared across several, the next install or update without a narrower row creates a separate secret for that deployment and supplies it to the OIDC socket. Coordinate consumer and provider reconciliation when rolling this change. To keep the old value, copy it to deployment scope first. To keep an explicit shared override, provision it at stack scope, or at environment scope for consumers in one environment. The tenant row is left untouched.
CLI operations
Section titled “CLI operations”shc vault set <key> <value> [--scope stack] [--stack blog]shc vault get <key> [--scope stack] [--stack blog]shc vault ls [--scope tenant]shc vault exists <key> [--stack blog]shc vault rm <key> [--scope stack] [--stack blog]These commands take --stack and --environment to select the scope
tuple. set, get, ls and rm also take --scope. Source:
modules/vault/commands/.
Common recipes
Section titled “Common recipes”# Replace an autogenerated admin passwordshc vault set admin_password "new-strong-password" \ --stack blog --environment prod
# Inspect everything at tenant scope for auditshc vault ls --scope tenant
# Copy a secret from one environment to anotherVAL=$(shc vault get api_key --stack myapp --environment staging)shc vault set api_key "$VAL" --stack myapp --environment prod
# Rotate, then restart dependent stacksshc vault set db_password "new-pw" --stack postgres --scope stackshc restart blog && shc restart analytics && shc restart wikiExternal providers
Section titled “External providers”SHC can delegate vault reads/writes to an external system. Each
provider is a type in
modules/vault/providers/
with Get, Set, Delete, Exists, List methods.
Supported providers
Section titled “Supported providers”| Provider | URI prefix | Source |
|---|---|---|
| Local (default, encrypted in the state database) | ref+vault:// | providers/local.go |
| AWS Secrets Manager | ref+aws:// | providers/aws.go |
| Azure Key Vault | ref+azure:// | providers/azure.go |
| GCP Secret Manager | ref+gcp:// | providers/gcp.go |
| HashiCorp Vault | ref+hashicorp:// | providers/hashicorp.go |
| Bitwarden | ref+bitwarden:// | providers/bitwarden.go |
| Doppler | ref+doppler:// | providers/doppler.go |
| Infisical | ref+infisical:// | providers/infisical.go |
| 1Password (Connect / CLI) | ref+onepassword:// | providers/onepassword.go |
pass (the standard Unix password manager) | ref+pass:// | providers/pass.go |
| SOPS | ref+sops:// | providers/sops.go |
Mixing providers in one cluster
Section titled “Mixing providers in one cluster”You can use multiple providers simultaneously: the URI scheme picks which provider resolves a reference. Typical pattern:
localfor app-generated secrets (autogen’d admin passwords, session keys): stays alive with the cluster, zero infra deps.- External provider for shared org credentials (GitHub tokens, SaaS API keys): values managed centrally, rotated outside SHC.
Configuring an external provider
Section titled “Configuring an external provider”Each provider has its own config shape, usually credentials + an
endpoint. Configure under vault.providers: in your user config.
Example for HashiCorp Vault:
vault: providers: hashicorp: enabled: true address: "https://vault.example.com:8200" token_file: "/etc/shc/vault-token" path_prefix: "secret/shc/"Then reference with the provider-specific prefix:
GITHUB_TOKEN: "ref+hashicorp://shared/github-token"Exact config keys per provider live in the provider’s *Config
struct, e.g. AWSConfig in providers/aws.go.
Operational concerns
Section titled “Operational concerns”Backups
Section titled “Backups”Local vault values are included in cluster-wide backups of the state database but protected by the same master password. Restoring a backup from a cluster with a different master password requires rotation before secrets become decryptable. See disaster-recovery.md for the restore procedure.
External-provider values are not SHC-backed; they live in the external system. SHC backs up only the references, and assumes the external system has its own DR story.
Every vault read/write emits an audit event
(modules/vault/audit.go).
See
architecture/security.md#audit-trail
and
architecture/modules/audit.md.
The actual secret value is never in the audit event, only the key, scope, operation, and actor.
Failure modes
Section titled “Failure modes”| Error | Cause | Fix |
|---|---|---|
X401012VLTINV | invalid vault password: the supplied password does not open the vault (unlock, rotate) | Reconfigure password_file / password_command, restart daemon |
X500961VLTCRP | vault crypto failure {op} {error}: the blob would not decrypt: wrong key, truncated ciphertext, bad base64 | Restore from backup; rotate the password back to the one the blob was written under |
X400980VLTFMT | vault crypto format failure: the stored value is not in the envelope format at all (a pre-envelope row, a botched import) | shc doctor run --category vault → vault.unreadable_secrets |
X503903VLTPWM | vault is locked: the operation needs an unlocked vault | shc vault unlock |
X503902VLTNAV | vault not available with {reason} {error}: locked, or an external provider is unreachable | Unlock, or check external provider health |
shc doctor run --category vault runs the vault module’s diagnoses:
vault.status (initialised / locked / unlocked state), vault.secrets
(flags rows with an empty encrypted value, the signature of a botched
import or migration), and vault.unreadable_secrets (counts rows SHC
can no longer decrypt (anything not in the envelope format) and FAILS
when there are any; such a row was written before v0.0.16 and its
plaintext is unreachable).
Rate limits & caching
Section titled “Rate limits & caching”External provider reads are debounced via the
DebouncedVaultProvider wrapper
(providers/debounce.go):
identical reads within a short window are coalesced. This protects
you from accidentally DOSing your paid secret manager on large
install runs.
Forgetting a provider
Section titled “Forgetting a provider”Disabling a provider while references to it still exist will cause
the next render to fail with X503902VLTNAV. Either migrate the
references first (shc vault set <key> <value> on a different
provider and update the YAMLs), or mark the provider read-only while
you migrate.
Security notes
Section titled “Security notes”- Never log secret values. SHC’s
logmodule scrubs fields typed as@secretwhen recording events, but don’t defeat this by stuffing secrets into non-@secretvariables. - Master password compromise ⇒ rotate immediately
(
shc vault rotate) and treat all previously-issued credentials as suspect. - Backups of
/var/lib/shccontain the encrypted vault. They are still protected by the master password, but treat them with appropriate care (encrypt at rest, limit access). - See security.md for the full threat model.
Related
Section titled “Related”- Architecture: security: vault in context
- Architecture: vault module: internals
- App definition: variables & secrets: app-author view
- Disaster recovery: restore flows that touch the vault
- Error reference: vault codes: all 40 of them, generated from the registry, among them
X401012VLTINV,X500961VLTCRP,X503903VLTPWM,X503902VLTNAV, …