Troubleshooting

First-stop index for when something isn’t working. Look up your symptom below; each entry points to the relevant page, source code, or error code.

For deep operational topics (upgrades, disaster recovery, backups), see ../operations/.

For the full list of every error code SHC can emit, see ../../reference/errors.md — 508 codes across 27 modules, auto-generated from source.

• Install fails before any container starts → Common install failures
• shc install hangs at “starting compose” → Compose-up failures
• “stack already exists” errors → X409003STKEXS
• Update rolled back with no clear reason → Upgrade flow — revert semantics
• Uninstall left resources behind → Uninstall flow — uninstall is best-effort forward, check logs for skipped steps

• “no matching sockets found” / X400903INVAPP → Common integration failures
• “multiple plugs match” → explicit integration spec → plugs-and-sockets — CLI syntax
• Plug connected but consumer can’t see variables → Compose template injection
• Integration phase job fails → X500009INTJOB, check shc job runs <stack> + provider container logs

• Quorum lost, writes returning 503 → Disaster recovery — Scenario 2
• Node stuck in “unhealthy” after crash → Disaster recovery — Scenario 1
• shc node join hangs → Cluster join flow — check mesh connectivity first (UDP 4242)
• Mesh join fails with cert errors → Disaster recovery — Scenario 5
• Clock-skew errors on join → fix with chrony / systemd-timesyncd; SHC rejects joins with skew > ~60s

• Backup creation fails → Common backup failures
• Restore doesn’t find the target stack → the stack is restored under the (tenant, stack, environment) tuple; check you’re targeting the right environment
• “Converter failed” during backup → X500908HKFAIL — run the converter manually to reproduce; see backup-restore flow

• “snapshots not supported on this filesystem” → Storage providers — picking a filesystem
• NFS mount keeps dropping → Storage providers — common issues
• Stack install fails: “no storage provider qualifies” → check the app’s requires.storage and the target node’s linked providers with shc storage ls

• Vault locked, writes failing → Vault — Unlocking on daemon start
• {{ vars.password }} renders as literal ref+vault://... → the vault couldn’t resolve the reference. Check scope, provider health, and Vault — Failure modes
• Restored backup has undecryptable secrets → master password from the source cluster differs; rotate (shc vault rotate-password)

• shc events disconnects / gaps → SSE troubleshooting
• No events at all → check shc status — a stuck queue shows tasks piling up without draining. See Observability
• Audit log empty → verify the actor had a populated Ctx — unauthenticated requests via localhost UDS skip audit by design; see security — audit trail

• “permission denied” on a valid-looking action → X403900NOPERM. See security — authorization for how scopes + resources match.
• “tenant not found” when you’re sure it exists → you’re in the wrong tenant. Check shc ctx / SHC_TENANT.

Modules register self-checks with the doctor framework; shc doctor run sweeps them all. Diagnoses only by default — repairs need an explicit --fix:

shc doctor run                       # run every registered diagnosis
shc doctor run --category storage    # only one module's diagnoses
shc doctor run --fix                 # also apply prescriptions for fixable findings
shc doctor ls                        # list registered diagnoses (and their categories)

Other high-signal commands:

shc node ls                          # cluster membership + leader
shc cluster status                   # node counts, voter/quorum state, leader
shc cluster mesh diagnose            # overlay reachability between every node pair
curl http://127.0.0.1:4003/health    # daemon health (JSON)
shc events                           # live lifecycle event stream
shc events --audit                   # recent audited actions
shc job runs <stack>                 # recent hook / integration job runs
shc logs tail                        # structured daemon log
shc hud                              # live TUI overview

• Look up the error code. Error reference has every one with source location.
• Check the flow doc. The flows pages trace each major pipeline step-by-step with file/line pointers — you can see exactly where you’re stuck.
• Read the source. SHC’s architecture docs cite source files and line numbers directly. Clone the repo, search for the error code, read the raising function.

• Common issues (detailed walkthroughs)
• SSE streaming troubleshooting