Skip to content
SHC Docs

shc doctor

diagnose and fix system issues

Reference for the doctor CLI group. Source: modules/doctor/commands/.

SubcommandDescription
runrun all diagnoses
listlist registered diagnoses

run all diagnoses

Diagnoses only by default: prescriptions (repairs) run only with an explicit --fix. The sweep runs on the daemon when one is reachable (that is where the database and keycloak checks are wired); with no daemon it degrades to a local diagnose-only pass over this host’s view, and --fix is refused.

On a cluster the run reports every node. Each cluster-scoped check runs once; each node-scoped check runs on every node and prints one row per node, the node’s name in the last column. A node that does not answer, or that the roster reads unreachable, prints one [skipped] doctor.node-checks row naming it and saying why. A node applies its own fixes: another node’s row carries its fix withheld with the command to run on that node. The exit code counts every node. Which checks are node-scoped is the SCOPE column of shc doctor ls, listed in the doctor module.

FlagTypeRequiredHelp
--fixboolnoapply prescriptions for fixable findings
--category / -cstrnorun only diagnoses in this category, or only the diagnosis with this name

A result is graded by its worst finding, at that finding’s own tier:

VerdictWhenExit
passno findings0
infothe worst finding is info: real observations that are not faults0
warnthe worst finding is warning3
failthe worst finding is error or critical1
fixed--fix ran and repaired every finding worse than info. One it left grades the result by its own tier0
skippedthe diagnosis could not run here (dependency unwired, seam absent) or it panicked0
inapplicablethe diagnosis reads an engine this node’s runtime does not run, such as a docker check on an incus node0

The run exits on its worst row, on any node: 1 when a diagnosis fails, 3 when the worst is a warning, 0 otherwise. 2 stays the usage error it is for every verb. A warn is a row you should act on, and a script that tests for exit 0 stops on it. An info row is one to read, not a break. A skipped is not a pass: it means the question was never asked. An inapplicable check has no question to ask on this node. It prints no finding and opens no condition. The run’s summary counts each verdict under its own key (passed, info, warnings, fixed, failed, skipped, inapplicable), so the counts add up to the checks run.

shc doctor run prints one generic invitation (“pass --fix to apply prescriptions”) for any run carrying a fixable finding, and that line is identical whether the prescription creates a directory or deletes a row. Before you type --fix, know which one you have:

DiagnosisWhat its prescription doesReversible?
system.directoriescreates the missing state directoriesn/a (creates only)
keycloak.realmsprovisions the missing realm, its shc-access client and the canonical roles (idempotent)n/a (creates only)
repo.sync-statere-syncs the repo with force=true, which refreshes a catalogue cache; re-renders no deploymentyes (re-sync)
database.stale-statusrewrites the desired word on a deployment row to agree with its status enumyes (a row value)
state.container-mismatchrewrites a deployment row’s status to degradedyes (a row value)
fabric.ingress-unitre-finishes stored deployments to free ingress units and moves the system Traefik onto each free unit, one node at a timeyes
database.orphaned-stacksDELETEs the deployment row of every deployment whose on-disk state is missingno

When shc doctor run -c fabric.ingress-unit --fix recreates a system Traefik whose package is newer in the catalogue, its output names shc update traefik -t master to take that package.

Everything else in the sweep is inspection-only and --fix cannot touch it, including app.package-drift, whose only conceivable repair is shc update <stack>, a re-render and restart of a running deployment. It carries no prescription on purpose (see Refreshing app packaging).

app.pinned-package-fetchable: can the pinned bytes still be fetched?

Section titled “app.pinned-package-fetchable: can the pinned bytes still be fetched?”

This one answers a question no other check asks, and its severity is worth understanding before you read a red line from it.

A deployment pinned to an exact package revision (shc ls shows app_version: 15.118+2) keeps running from bytes that were fetched once and extracted into a node-local cache. Every other signal in this platform is satisfied by that cache: app.package-drift resolves the deployment’s package happily from it, without asking the registry anything. So when the registry deletes the pinned revision, nothing notices: the stack serves, shc ls is green, tofu plan says no changes, and the pin only fails the next time something has to fetch it again: a restore, a node move, a refresh_app_package, or a re-render on a node that never held the cache. Those are the operations you run when you are already in trouble.

The check reports four distinct populations, and they are not the same fault:

FindingWhat it meansSeverity
pinned to a revision the catalogue no longer publishesretention deleted it. Establishable with no network: it is a statement about the catalogue this node already holds, so an offline estate gets it at full strengtherror
pinned to a revision the catalogue advertises and cannot servethe index and the registry disagree, a publisher fault that also affects unpinned deployments resolving latesterror
could not be reached / could not be examinedretryable (5xx, 429, DNS), or no catalogue and no probe: the normal state of an airgapped estate, which is not a faultwarning / info
not probed in this runthe node’s own rate limiter for the registry had no turn left for the probe before the check’s deadline, so nothing was sent. It says nothing about the registry or the route to itinfo

A bare-line pin (18.11) is judged by the newest revision of that line in the catalogue, the same revision install and update select. A full pin (18.11+25) matches only that revision; a line with no published revision produces the first finding above.

It prescribes nothing, and --fix cannot touch it. There is no bookkeeping repair for an unfetchable pin; there are three real options and two of them are not the platform’s to take:

  1. Republish the pinned revision to the registry. Done off the estate, in the publisher. Changes nothing running, and it is the only option that keeps the deployment’s own bytes.
  2. Re-pin forward: shc update <stack> --app-version <v>. Mutates production: re-renders and restarts the deployment, and silently adopts every packaging change between the two revisions. Compare them first; the catalogue’s per-revision images list is the diff that matters.
  3. Accept the risk deliberately. The stack keeps serving; a restore, a node move or a refresh will fail until (1) or (2) happens.

Nothing in SHC will rewrite the pin, delete the row that records it, or otherwise “repair” this for you.

dns.record-ownership: which DNS records does this estate own?

Section titled “dns.record-ownership: which DNS records does this estate own?”

Every managed record SHC publishes gets a companion ownership TXT at _shc-owner.<host> reading heritage=shc,shc/owner=<this cluster's id>. That marker is what lets the platform sweep its own residue without ever touching the apex, the MX, the SPF/DKIM/DMARC rows or another controller’s records: they carry no claim, so they are structurally invisible to it.

Until this check existed, the marker was invisible to operators too: answering “which records does this estate own, and is any of them stale?” meant digging each host by hand. There is deliberately no shc dns verb: records are lifecycle-managed (--expose), and the CLI tree was removed by ruling, so the read arrives here, as a diagnosis over the same cluster-wide sweep the orphan reaper runs.

FindingWhat it meansSeverity
not in its desired-host setthis cluster’s claim on a host it no longer intends to publish. Residue from an estate terraform destroy, a superseded system host, or an uninstall that never released, and a record pointing at an address the cloud is free to recycle to somebody elsewarning
claimed but holds no A/AAAA/CNAMEthe host resolves nowhere. The record was removed outside SHC and the claim outlived it; nothing republishes it on a timerwarning
claimed by another ownera host this cluster intends to publish whose records belong to somebody else. Every sync of it is skipped (X403040DNSOWN), so it can never converge. This is the shape that took SSO down on a rebuildwarning
a previous build of this cluster’s identitya destroy+rebuild re-minted the per-instance owner id, and the desired-host set cannot prove these hosts ours. Never swept, only reported (X403044DNSRPL)warning
zones that could not be enumeratedcredentials drifted or the API refused. Those zones contribute nothing to the counts, so the run’s silence about them is not an all-clearwarning
pinned claimsshc/pinned=1, minted by terraform’s shc_dns_record. Exempt from the orphan reaper permanently, so they survive an uninstall and are released only by an explicit deleteinfo
live records carrying no claima delete would refuse them (X403041DNSUNO), so they outlive the deployment, but the next managed sync self-adopts the claim and the state clears itselfinfo
no zone was reachablenothing at all was established about this estate’s recordsinfo

It prescribes nothing, and --fix cannot touch it. That is by design, and the design is pinned by tests in both repos. There is no bookkeeping repair here: every remedy is a real change to a live zone or a re-render of a running deployment.

  1. Sweep the orphans: POST /api/dns/reap with ?apply=true. Mutates a live zone. It refuses when the orphan set is a majority of what the cluster owns, because that is the shape a partial desired-host read produces. Read the list first: this pass has already been wrong once, and a record that looks stale from one node’s view may be a live host.
  2. Republish a dark host: shc update <stack>. Mutates production: re-renders and restarts the deployment, which is what recreates the record.
  3. Settle a foreign or predecessor claim by hand at the provider. SHC will not take a claim it cannot prove is its own; that refusal is the only thing stopping two clusters sharing a zone from deleting each other’s records.

It says nothing about mail DNS: the apex, MX, SPF, DKIM and DMARC carry no ownership claim, which is exactly why the sweeps cannot delete them. Publishing mail records is unbuilt and its ownership disposition is still open; see the DNS verb question.

database.orphaned-stacks deserves the extra look. Its finding text names a missing directory; its repair is an irreversible row delete, and it has already been wrong once: a vapp instance legitimately has no deployment directory of its own (its record is a sidecar under the parent), so the check went red on the leader for twelve healthy live deployments. That was fixed, but the shape of the risk is permanent: a delete driven by a stat is only ever as right as the path it stats. Read the findings before you fix them.

Where a findings hint points at a command that changes running state, the hint leads with what it costs: Mutates production when a stack will be re-rendered and restarted, Read-only when it will not. This is a contract, not a style preference: a doctor line whose advice quietly takes a service down is worse than no line at all, because the next one gets ignored too.

list registered diagnoses

Lists every diagnosis registered with the doctor framework, useful for sanity-checking module wiring and authoring --category filters.

The table has five columns: NAME, CATEGORY, SCOPE (cluster when shc doctor run answers the check once, node when every node answers it), FIX (the declared class of the diagnosis’s prescription, - when it has none) and DESCRIPTION, one line per diagnosis saying what it reads. Every diagnosis carries a description; the registry refuses one without. The remedy is not a default column: shc doctor ls --fields name,remedy prints it as the class and the effect, for example destructive: DELETE FROM deployment WHERE id = ?, one row per flagged deployment, permanently removing the control plane's record of it, and -o json carries the whole declaration (class, effect, reversibility, consequence) as an object.

Source: modules/doctor/commands/