Skip to content
SHC Docs

Health module

Layer 11. Lifecycle. Lifecycle It reconciles a running deployment.

Source: modules/health/.

CLIshc status
HTTP endpointssee the route reference
Cross-module depsapp, backup, docker, job, metrics, network, node, vault
modules/health/
├── commands/
├── events.go
├── poll_deployments.go
├── poll_nodes.go
├── router.go # /health, /ready endpoints
├── state.go # module-level HealthState singleton
├── tasks.go
└── types.go # HealthState, NodeHealth, DeploymentHealth

Three concerns, all rooted in the health module:

  1. Runtime health endpoints: liveness, readiness, and a detailed health report for load balancers, shc status, and monitoring.
  2. Cron tiers: the structured periodic work that polls the cluster, reconciles degraded deployments, and purges old rows.
  3. The HealthState singleton: the module-level snapshot of cluster state that crons write to and the HUD / endpoints read from.

The module owns no subsystem lifecycle; HealthState is a plain module-level object decoupled from subsystem start/stop.

All periodic work runs through three leader-only cron tiers plus a set of DB-tracked maintenance cron jobs. Every cron uses skip-if-running: if the previous invocation has not completed, the next tick is silently skipped.

CronTypeDefaultConfig path
Tickrpc cron15 stiming.tick_interval
Pollrpc cron1 mintiming.poll_interval
Reconcilerpc cron1 mintiming.reconcile_interval
Cleanuptask cron1 hmaintenance.cleanup.interval
Disk scantask cron1 hmaintenance.disk_scan.interval
Usage reconciletask cron6 hmaintenance.usage_reconcile.interval
Disk auto-prunetask cron1 hmaintenance.disk_auto_prune.interval
Repo synctask cron6 hmaintenance.repo_sync.interval
Retention cleanuptask crondaily 02:00maintenance.retention_cleanup.schedule

All intervals are config-driven via function references resolved at scheduler start; see cron in the communication-patterns doc.

Fast, lightweight data gathering. Runs sequentially within one tick:

  1. node_metrics: broadcast gather, each node returns system metrics; leader writes to HealthState.
  2. container_metrics: broadcast gather, each node returns docker stats.
  3. global_diagnostics: local leader query for task/worker/cron counts.
  4. quota_metrics: local leader collection of quota gauges.
  5. vault_check: broadcast fire-and-forget; if any node reports locked and the leader holds the password in memory, re-broadcast unlock (see Vault unlock flow).

Moderate-weight state sync:

  1. deployment_sync: broadcast docker compose ps to every node; leader writes container/service state to DB and to HealthState.
  2. retention_config_check: leader-local audit of retention policies.

The deployment rollup: what “healthy N/N” actually claims

Section titled “The deployment rollup: what “healthy N/N” actually claims”

deployment_sync (leader) and the follower-side read-path derivation (Service.Status when no local poll has run) share ONE fold: deriveOneDeployment in modules/health/derive.go. The evidence it consults, per container:

FactSourceUsed for
statusrecorder-written container row (raft-replicated)readiness
healthsamereadiness; blank ⇒ unprobed
SHC container statusderived at the seam from exit code + restart policya restart “no” service that ended 0 counts as ready
exit_cause / oom_kill_count / oom_memory_limit_bytessame rowthe reason line
updated_atsame rowstaleness (a frozen row = a gone container)
RestartCount / State.StartedAtnode-local metrics.StatsCache (2 s tick, fanned in cluster-wide)liveness

The last row is the newest and closes a measured defect: shc status reported mailcow healthy 18/18 while watchdog-mailcow had restarted 117 times. Every fact above the line describes the container at an instant, and a crash-looping container is “Up” most instants, so a loop and a steady process were indistinguishable to the fold. (The daemon was not ignorant of it: the reason column read watchdog-mailcow stopped, cause unknown beside the healthy verdict.)

The rule (modules/health/restart_signal.go, IsFlapping):

a container is FLAPPING when RestartCount >= 3 and now - State.StartedAt < 10m.

Why both halves:

  • RestartCount is cumulative and never resets, so it alone cannot say whether the restarts are happening now: a container with 117 lifetime restarts that has been solid for a week would stay red forever, and a permanently-red signal is one nobody reads.
  • State.StartedAt supplies the “now” without any remembered history. For a container under a restart policy the age of the current run is the observed inter-restart interval. A window-delta rate would need samples, so it would be blind after every daemon restart and every leader change, and it could not see a loop with a period longer than the window.
  • 3 excludes the ordinary deploy race (an app losing the connect to its database once or twice before it settles), the most common restart shape on a healthy estate. Docker’s restart backoff doubles from 100 ms, so three restarts is under a second of backoff: looping, not recovering.
  • 10 min is longer than a slow-but-legitimate start (a database doing crash recovery) and short enough that “flapping” means now. It is also the self-clearing mechanism: nothing resets RestartCount, so the settle window is the only thing that can return a deployment to healthy.

Effect on the verdict. A flapping service is excluded from services_healthy (mailcow now reads 17/18, not 18/18), and the reason names it: watchdog-mailcow restarting repeatedly (117 restarts, up 8s). A deployment whose only fault is flapping is clamped to degraded, never unhealthy; unhealthy means every container is down, and a looping container is up part of the time. When something is genuinely down alongside it, unhealthy stands.

It fails open. No cache entry (cold daemon, collectors not warm, a peer whose fan-in has not landed) or a zero StartedAt means no verdict, and the rollup answers exactly as it did before. The signal does not feed the reconciler: ReconcileDegradedStacks reads the deployment row status via ListBrokenDeployments, not HealthState, so a flapping verdict cannot drive an auto-restart loop that would feed itself more restarts.

Unprobed services: a running-state claim is not a liveness claim

Section titled “Unprobed services: a running-state claim is not a liveness claim”

A RUNNING container whose observer reports no health token declares no healthcheck; docker emits healthy / unhealthy / health: starting for every container that has one. It has therefore passed nothing. Measured on huly: 12/12 healthy where EIGHT of the twelve services declared no probe at all.

The rollup now counts them (services_unprobed on the wire, omitted when zero) and shc status names them under the table:

3 services counted healthy declare no healthcheck; that is a
running-state claim, not a liveness claim: huly (8), plane (2), ...

The verdict is deliberately unchanged. Subtracting unprobed services from services_healthy is the strictly honest fold and it would flip a large share of both production estates to degraded in one release, for a condition no operator can fix from the status surface; the fix is a healthcheck in the app (as huly’s was). Publishing the number puts the pressure where it can be acted on without an alarm storm.

Note also that a probe existing is not sufficient: huly’s original socket-listening check reported LISTENING for every SIGSTOP-frozen process, so “the socket is open” is not liveness either. The unprobed count measures probe presence, not probe quality.

Self-healing for degraded deployments plus infrastructure enforcement. Broadcast fire-and-forget; each node acts on its own local state.

Per-node flow:

  1. Read in-memory HealthState for this node’s deployments.
  2. If any deployment is degraded, do a live docker compose ps re-check before acting; never reconcile on stale in-memory state alone.
  3. If still degraded:
    • Skip if already in the resolving set for this deployment.
    • Skip if cooldown has not elapsed since last attempt.
    • Apply exponential backoff on repeated failures.
    • Add to resolving set → run docker compose up -d → remove on completion.
  4. All cooldown / backoff state is in-memory only and resets on process restart.

Infrastructure enforcement (same cron): Traefik port-binding check and restart, OpenObserve health check, docker network creation.

Run via task cron, so each invocation creates a TaskModel row and is audit-tracked.

The Cleanup job also does task purging:

  • Completed tasks older than maintenance.completed_task_retention (default 24 h) are bulk-deleted.
  • Failed tasks older than maintenance.failed_task_retention (default 7 d) are bulk-deleted.

This is a single bulk DELETE; the retention defaults keep failed rows longer for debugging.

Lives at modules/health/state.go as a module-level instance of the HealthState struct:

type HealthState struct {
Nodes map[string]*NodeHealth
Deployments map[string]*DeploymentHealth
LastPoll time.Time
}

Writers: the Tick and Poll crons (node_metrics, container_metrics, deployment_sync). Readers: the HUD collector, shc status, and the detailed shc.status.get method.

Alongside HealthState, the same module holds:

  • _cron_heartbeats: dict[str, datetime]: last successful completion per cron (read by the HUD for heartbeat display).
  • Reconcile backoff state (cooldown dict, resolving set) also in-memory only.

Unlock: user POSTs to shc.vault.unlock with the password:

  1. Request is routed to the leader (existing behavior).
  2. Leader calls unlock_vault(password) locally; password held in memory only.
  3. Leader calls the vault.unlock RPC function on every other node over the mTLS internal API.
  4. Each node unlocks locally.

Ongoing enforcement runs from the Tick cron’s vault_check step: leader broadcasts a vault-status check; any node reporting locked triggers a re-broadcast of the unlock if the leader still holds the password. On leader restart, the leader first tries the configured env var / password manager; if neither yields a password, nodes stay locked until a human unlocks again.

Security constraints:

  • There is no endpoint to read the vault password, only write (unlock).
  • The password is only transmitted over the mTLS internal API (port 4003).
Metric1 node10 nodes50 nodes
Tick broadcast RPCs / min4 (self)40200
Poll broadcast RPCs / min11050
Reconcile broadcast RPCs / min11050
Internal-API calls / min (total)~6~60~300

Broadcasts run concurrently through goroutines and errgroup in RpcTransport.FanOut; 50 nodes = 5 calls/s, comfortably within mTLS connection capacity.