shc doctor
diagnose and fix system issues
Reference for the doctor CLI group. Source: modules/doctor/commands/.
Subcommands
Section titled “Subcommands”| Subcommand | Description |
|---|---|
run | run all diagnoses |
list | list registered diagnoses |
shc doctor run
Section titled “shc doctor run”run all diagnoses
Diagnoses only by default: prescriptions (repairs) run only with an explicit --fix. The sweep runs on the daemon when one is reachable (that is where the database and keycloak checks are wired); with no daemon it degrades to a local diagnose-only pass over this host’s view, and --fix is refused.
On a cluster the run reports every node. Each cluster-scoped check runs once; each node-scoped check runs on every node and prints one row per node, the node’s name in the last column. A node that does not answer, or that the roster reads unreachable, prints one [skipped] doctor.node-checks row naming it and saying why. A node applies its own fixes: another node’s row carries its fix withheld with the command to run on that node. The exit code counts every node. Which checks are node-scoped is the SCOPE column of shc doctor ls, listed in the doctor module.
| Flag | Type | Required | Help |
|---|---|---|---|
--fix | bool | no | apply prescriptions for fixable findings |
--category / -c | str | no | run only diagnoses in this category, or only the diagnosis with this name |
Verdicts and exit code
Section titled “Verdicts and exit code”A result is graded by its worst finding, at that finding’s own tier:
| Verdict | When | Exit |
|---|---|---|
pass | no findings | 0 |
info | the worst finding is info: real observations that are not faults | 0 |
warn | the worst finding is warning | 3 |
fail | the worst finding is error or critical | 1 |
fixed | --fix ran and repaired every finding worse than info. One it left grades the result by its own tier | 0 |
skipped | the diagnosis could not run here (dependency unwired, seam absent) or it panicked | 0 |
inapplicable | the diagnosis reads an engine this node’s runtime does not run, such as a docker check on an incus node | 0 |
The run exits on its worst row, on any node: 1 when a diagnosis fails, 3 when
the worst is a warning, 0 otherwise. 2 stays the usage error it is for every
verb. A warn is a row you should act on, and a script that tests for exit 0
stops on it. An info row is one to read, not a break. A skipped is not a
pass: it means the question was never asked. An inapplicable check has no
question to ask on this node. It prints no finding and opens no condition. The
run’s summary counts each verdict under its own key (passed, info,
warnings, fixed, failed, skipped, inapplicable), so the counts add
up to the checks run.
What --fix actually changes
Section titled “What --fix actually changes”shc doctor run prints one generic invitation (“pass --fix to apply
prescriptions”) for any run carrying a fixable finding, and that line is
identical whether the prescription creates a directory or deletes a row. Before
you type --fix, know which one you have:
| Diagnosis | What its prescription does | Reversible? |
|---|---|---|
system.directories | creates the missing state directories | n/a (creates only) |
keycloak.realms | provisions the missing realm, its shc-access client and the canonical roles (idempotent) | n/a (creates only) |
repo.sync-state | re-syncs the repo with force=true, which refreshes a catalogue cache; re-renders no deployment | yes (re-sync) |
database.stale-status | rewrites the desired word on a deployment row to agree with its status enum | yes (a row value) |
state.container-mismatch | rewrites a deployment row’s status to degraded | yes (a row value) |
fabric.ingress-unit | re-finishes stored deployments to free ingress units and moves the system Traefik onto each free unit, one node at a time | yes |
database.orphaned-stacks | DELETEs the deployment row of every deployment whose on-disk state is missing | no |
When shc doctor run -c fabric.ingress-unit --fix recreates a system Traefik whose package is newer in the catalogue, its output names shc update traefik -t master to take that package.
Everything else in the sweep is inspection-only and --fix cannot touch it,
including app.package-drift, whose only conceivable repair is
shc update <stack>, a re-render and restart of a running deployment. It carries no
prescription on purpose (see
Refreshing app packaging).
app.pinned-package-fetchable: can the pinned bytes still be fetched?
Section titled “app.pinned-package-fetchable: can the pinned bytes still be fetched?”This one answers a question no other check asks, and its severity is worth understanding before you read a red line from it.
A deployment pinned to an exact package revision (shc ls shows
app_version: 15.118+2) keeps running from bytes that were fetched once and
extracted into a node-local cache. Every other signal in this platform is
satisfied by that cache: app.package-drift resolves the deployment’s package
happily from it, without asking the registry anything. So when the registry
deletes the pinned revision, nothing notices: the stack serves, shc ls is
green, tofu plan says no changes, and the pin only fails the next time
something has to fetch it again: a restore, a node move, a
refresh_app_package, or a re-render on a node that never held the cache.
Those are the operations you run when you are already in trouble.
The check reports four distinct populations, and they are not the same fault:
| Finding | What it means | Severity |
|---|---|---|
| pinned to a revision the catalogue no longer publishes | retention deleted it. Establishable with no network: it is a statement about the catalogue this node already holds, so an offline estate gets it at full strength | error |
| pinned to a revision the catalogue advertises and cannot serve | the index and the registry disagree, a publisher fault that also affects unpinned deployments resolving latest | error |
| could not be reached / could not be examined | retryable (5xx, 429, DNS), or no catalogue and no probe: the normal state of an airgapped estate, which is not a fault | warning / info |
| not probed in this run | the node’s own rate limiter for the registry had no turn left for the probe before the check’s deadline, so nothing was sent. It says nothing about the registry or the route to it | info |
A bare-line pin (18.11) is judged by the newest revision of that line in the
catalogue, the same revision install and update select. A full pin (18.11+25)
matches only that revision; a line with no published revision produces the first
finding above.
It prescribes nothing, and --fix cannot touch it. There is no bookkeeping
repair for an unfetchable pin; there are three real options and two of them are
not the platform’s to take:
- Republish the pinned revision to the registry. Done off the estate, in the publisher. Changes nothing running, and it is the only option that keeps the deployment’s own bytes.
- Re-pin forward:
shc update <stack> --app-version <v>. Mutates production: re-renders and restarts the deployment, and silently adopts every packaging change between the two revisions. Compare them first; the catalogue’s per-revisionimageslist is the diff that matters. - Accept the risk deliberately. The stack keeps serving; a restore, a node move or a refresh will fail until (1) or (2) happens.
Nothing in SHC will rewrite the pin, delete the row that records it, or otherwise “repair” this for you.
dns.record-ownership: which DNS records does this estate own?
Section titled “dns.record-ownership: which DNS records does this estate own?”Every managed record SHC publishes gets a companion ownership TXT at
_shc-owner.<host> reading heritage=shc,shc/owner=<this cluster's id>. That
marker is what lets the platform sweep its own residue without ever touching
the apex, the MX, the SPF/DKIM/DMARC rows or another controller’s records:
they carry no claim, so they are structurally invisible to it.
Until this check existed, the marker was invisible to operators too:
answering “which records does this estate own, and is any of them stale?” meant
digging each host by hand. There is deliberately no shc dns verb: records
are lifecycle-managed (--expose), and the CLI tree was removed by ruling, so
the read arrives here, as a diagnosis over the same cluster-wide sweep the
orphan reaper runs.
| Finding | What it means | Severity |
|---|---|---|
| not in its desired-host set | this cluster’s claim on a host it no longer intends to publish. Residue from an estate terraform destroy, a superseded system host, or an uninstall that never released, and a record pointing at an address the cloud is free to recycle to somebody else | warning |
| claimed but holds no A/AAAA/CNAME | the host resolves nowhere. The record was removed outside SHC and the claim outlived it; nothing republishes it on a timer | warning |
| claimed by another owner | a host this cluster intends to publish whose records belong to somebody else. Every sync of it is skipped (X403040DNSOWN), so it can never converge. This is the shape that took SSO down on a rebuild | warning |
| a previous build of this cluster’s identity | a destroy+rebuild re-minted the per-instance owner id, and the desired-host set cannot prove these hosts ours. Never swept, only reported (X403044DNSRPL) | warning |
| zones that could not be enumerated | credentials drifted or the API refused. Those zones contribute nothing to the counts, so the run’s silence about them is not an all-clear | warning |
| pinned claims | shc/pinned=1, minted by terraform’s shc_dns_record. Exempt from the orphan reaper permanently, so they survive an uninstall and are released only by an explicit delete | info |
| live records carrying no claim | a delete would refuse them (X403041DNSUNO), so they outlive the deployment, but the next managed sync self-adopts the claim and the state clears itself | info |
| no zone was reachable | nothing at all was established about this estate’s records | info |
It prescribes nothing, and --fix cannot touch it. That is by design, and the
design is pinned by tests in both repos. There is no bookkeeping repair here:
every remedy is a real change to a live zone or a re-render of a running deployment.
- Sweep the orphans:
POST /api/dns/reapwith?apply=true. Mutates a live zone. It refuses when the orphan set is a majority of what the cluster owns, because that is the shape a partial desired-host read produces. Read the list first: this pass has already been wrong once, and a record that looks stale from one node’s view may be a live host. - Republish a dark host:
shc update <stack>. Mutates production: re-renders and restarts the deployment, which is what recreates the record. - Settle a foreign or predecessor claim by hand at the provider. SHC will not take a claim it cannot prove is its own; that refusal is the only thing stopping two clusters sharing a zone from deleting each other’s records.
It says nothing about mail DNS: the apex, MX, SPF, DKIM and DMARC carry no ownership claim, which is exactly why the sweeps cannot delete them. Publishing mail records is unbuilt and its ownership disposition is still open; see the DNS verb question.
database.orphaned-stacksdeserves the extra look. Its finding text names a missing directory; its repair is an irreversible row delete, and it has already been wrong once: a vapp instance legitimately has no deployment directory of its own (its record is a sidecar under the parent), so the check went red on the leader for twelve healthy live deployments. That was fixed, but the shape of the risk is permanent: a delete driven by astatis only ever as right as the path it stats. Read the findings before you fix them.
A prescription that mutates says so
Section titled “A prescription that mutates says so”Where a findings hint points at a command that changes running state, the hint
leads with what it costs: Mutates production when a stack will be re-rendered
and restarted, Read-only when it will not. This is a contract, not a style
preference: a doctor line whose advice quietly takes a service down is worse than
no line at all, because the next one gets ignored too.
shc doctor list
Section titled “shc doctor list”list registered diagnoses
Lists every diagnosis registered with the doctor framework, useful for sanity-checking module wiring and authoring --category filters.
The table has five columns: NAME, CATEGORY, SCOPE (cluster when shc doctor run answers the check once, node when every node answers it), FIX (the declared class of the diagnosis’s prescription, - when it has none) and DESCRIPTION, one line per diagnosis saying what it reads. Every diagnosis carries a description; the registry refuses one without. The remedy is not a default column: shc doctor ls --fields name,remedy prints it as the class and the effect, for example destructive: DELETE FROM deployment WHERE id = ?, one row per flagged deployment, permanently removing the control plane's record of it, and -o json carries the whole declaration (class, effect, reversibility, consequence) as an object.
Source: modules/doctor/commands/