Common issues

Detailed, symptom-first walkthroughs. For quick symptom→page routing see the troubleshooting index. For every error code’s exact definition see the error reference.

Install aborted validating the app definition. Causes, in rough order of likelihood:

1. Required variable not provided. The app declares vars.admin_password@secret! but you didn’t pass --var admin_password=.... Check which fields are ! (required) in the app’s vars: block or vars.yaml.
2. Required plug has no viable socket. An app with required: true on a plug refuses to install unless a matching socket exists. Either install the provider stack first, or pass -i <provider>.
3. Integration spec doesn’t resolve. Error detail says “no matching sockets found” or “multiple plugs match” — use the explicit -i plug=provider.socket form. See plugs-and-sockets — CLI spec.
4. Malformed app.yaml or vars.yaml. YAML parse error or a field of the wrong type. Run shc validate <path-to-app-dir> to get a precise schema error.

A stack with that (tenant, stack, environment) tuple is already installed.

shc stack list --all-tenants    # find the conflicting stack
shc uninstall <stack> -e <env>  # remove it, OR…
shc install <app> -s <name> -e <env>  # pick a different stack/env

Install reached compose up but containers exit. The install flow waits for a healthcheck window (configurable per-app via timeouts.start) and then rolls back.

1. shc stack logs <stack> — raw container logs
2. shc logs tail --filter stack=<stack> — daemon lifecycle events
3. shc -s <stack> events — structured events
4. Check X500907STKFAL detail in the logs — it includes which service’s healthcheck failed

Common root causes:

• Missing integration variable: template rendered DATABASE_URL=postgres://user:@host/db (empty password) → provider integration hadn’t finished when compose started. Re-run with -i <provider> to wire the integration, or install the provider first.
• Image architecture mismatch: running on arm64 but the app’s image is amd64-only. Check requires.architectures in app.yaml and the image’s manifest.
• Port already in use: something else on the host is bound to a public_ports target. ss -tlnp | grep <port>.
• Volume mount failed: the storage provider refused the mount path — check shc doctor run --category storage.

Docker pull is slow or unreachable from the target node.

# On the target node:
docker pull <image>                          # reproduce directly
shc node ls -v                               # find which node it targets
shc inspect deployment <stack>               # see assignment

If pulling works manually but SHC’s pull hangs, the issue is usually Docker daemon health. Restart Docker on the node.

The app name isn’t in any registered repo.

shc repo list                     # check repos are configured
shc repo sync                     # force-refresh
shc search <partial-name>         # confirm the app exists

If it’s a local file path repo, make sure shc has read access to the directory.

Auto-match failed because the consumer’s plugs don’t match any of the provider’s socket names.

# See the consumer's plugs:
shc inspect app <app>      # lists plugs + their target socket names

# See the provider's sockets:
shc inspect stack <provider-stack>

If the names don’t match (e.g. your plug wants postgres but the provider exports database), use explicit syntax:

shc install myapp -i postgres=provider-stack.database

Plug or socket name in your -i spec doesn’t exist on the named stack. Typo or stale spec. Verify names with shc inspect stack.

The integration didn’t register before compose rendered, so the template variable was unset. This almost always means the integration phase job failed silently.

shc integration list <stack>            # verify the edge exists
shc job runs <stack>                    # find the failed integration-phase job
shc job show <run-id>                   # get full stdout/stderr

Re-run the integration:

shc integration delete <stack> <plug>   # remove the broken edge
shc integration create <stack> <plug>=<provider>.<socket>

The provider’s socket has local_only: true, meaning it can only serve consumers on the same node. Your consumer is on a different node.

Options:

• Install the consumer on the same node (--node <same-as-provider>)
• Remove the local_only restriction in the provider app’s socket YAML (only appropriate if the exposed service is actually mesh-reachable)

A pre_backup or post_backup hook (or a backup converter) failed. SHC logs the job error but does not abort the backup — so this is a warning, not a fatal error, unless the backup evidently lacks expected content.

shc job runs <stack>                    # find the failed hook run
shc job show <run-id>                   # full output

Then:

• Run the job manually against the same stack to reproduce
• Check if the job expects a service that’s not yet running (run it at a different hook phase — see lifecycle-hooks — allowed contexts per hook)

Backup failed before capture because restic couldn’t access its repo.

1. Confirm backup.repo in your config points at a reachable location
2. For S3/B2 backends, confirm the backup.password reference resolves — test with shc vault get <ref>
3. For NFS/local shared repos, confirm the mount is present on every node (the node doing the backup is the node the stack lives on). See storage-providers — NFS.

Restore completed (metadata, volumes, vault secrets, integrations) but the restored stack’s containers fail to come up.

Most common causes:

• Integrations didn’t replay. Restore re-creates integration rows but may skip re-running the integration phase jobs if the provider is unavailable at restore time. Check shc integration list <stack> — if edges exist but wait_for jobs aren’t present in provider DBs, manually re-run:

  shc integration recreate <stack>

• Vault scope mismatch. Restoring into a different tenant or environment means ref+vault://...?scope=deployment references resolve to nothing until you set values in the new scope. See vault — scopes.
• Image version drift. The app was installed at v1 but now resolves to v2, and the restored data is v1-compatible. Pin versions in app.yaml or use a repo at a fixed ref.

The joining node can’t reach the leader’s enrollment API (TCP 4003) or can’t complete the mesh handshake (UDP 4242).

# On the joining node:
curl -v https://<leader-ip>:4003/health   # should succeed (mTLS error is OK - means TCP works)
nc -zvu <leader-ip> 4242                  # UDP reachability test

# On the leader:
ss -tlnp | grep 4003                       # internal API listening
ss -ulnp | grep 4242                       # Nebula listening

See cluster join flow for the full handshake.

Either the node is actually dead (see disaster recovery — Scenario 3) or the healthcheck probe can’t reach it.

# On any healthy voter:
shc cluster status                          # node counts, voter/quorum state, leader
shc cluster ping <node>                     # probe that node directly
shc cluster mesh diagnose                   # overlay reachability, every node pair

# Run this check from the "unhealthy" node itself if you have shell:
shc node ls                                 # if it also sees itself as lonely,
                                            # mesh is broken from its side

Raft lost quorum. Recover per disaster recovery — Scenario 2.

Logs show cert expiry warnings or rotation errors.

shc cluster cert ls                         # report cert ages across the cluster
shc cluster cert rotate                     # force-rotate everything that's due
shc cluster cert rotate --type mtls         # rotate only one category

See disaster recovery — Scenario 5.

X503902VLTNAV at startup. The daemon tried to unlock the vault and found no password in env, file, or command.

Configure one of:

1. SHC_VAULT_PASSWORD env var (best for systemd LoadCredential)
2. vault.password_file in config
3. vault.password_command in config

See vault — unlocking.

The stored blob can’t be decrypted with the current master password.

• You recently rotated and the rotation didn’t finish → restart daemon, retry
• You restored a backup from a different cluster → rotate to the backup’s original password, then rotate again to your new one
• Hardware corruption of the state database (/var/lib/shc/shc.db, or /var/lib/shc/rqlite in cluster mode) → restore from a clean backup

The resolver returned None for the key. Checks:

1. Does the value exist? shc vault exists <key> at the exact (scope, tenant, stack, environment) you expect.
2. Is the scope in the URI what you think? ?scope=tenant vs ?scope=deployment resolve against different keys.
3. Is the vault healthy? shc doctor run --category vault reports lock state and flags broken (empty-value) rows. An unreachable external provider surfaces as X503902VLTNAV (vault not available) in the daemon log.
4. Did you autogen? ?autogen only generates on first render; if the first render failed (vault write error), the key may still be absent on retry. Check for X500901OPFAIL in logs.

# 1. Is there actually a process?
systemctl status shc

# 2. What did it log?
journalctl -u shc -n 200

# 3. Can it reach its state dir?
ls -la /var/lib/shc/
sudo -u shc test -w /var/lib/shc && echo OK

# 4. Are required ports free?
ss -tlnp | grep -E '4001|4002|4003'
ss -ulnp | grep 4242

# 5. Is the raft engine's data corrupt? (cluster mode)
ls -la /var/lib/shc/rqlite/
# Look for a fresh 'raft.db' and old logs. If this is a cluster
# member whose peers are gone, see disaster-recovery Scenario 2.

• “address already in use” — another process owns the port. Find it with ss -tlnp. SHC’s internal API is TCP 4003 by default; change via internal_api.port in config.
• “permission denied” on /var/lib/shc/... — systemd user/group mismatch. Verify with stat /var/lib/shc and id shc. The state dir must be owned by the shc user.
• “vault password not configured” — see vault section above.
• “raft: failed to open snapshot” — the raft engine’s snapshot is corrupt. See disaster recovery — Option C: restore from backup.

If none of the above helps, turn up the logging and re-run:

# Run the daemon in foreground with debug logging:
systemctl stop shc
sudo -u shc SHC_LOG_LEVEL=debug shc daemon

…and inspect the full structured log (not just human messages) via shc logs tail (on a separate shell) while reproducing the issue.

For production incidents, gather the diagnostic surfaces into one place before opening an issue:

shc doctor run -o json    > doctor.json     # every registered diagnosis
shc version -o json       > version.json    # version + build info
shc cluster status -o json > cluster.json   # voter/quorum state, leader
shc node ls -o json       > nodes.json      # membership + node health
shc events --audit -n 200 > audit.log       # recent audited actions
journalctl -u shc --since -2h > daemon.log  # structured daemon log

• Error reference — every error code SHC emits
• Disaster recovery — cluster-level failures
• Observability — logs, metrics, traces
• Flow docs — step-by-step pipeline traces